Skip to content

[improvement](be) Add disk cache for external file metadata #63376

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
xylaaaaa wants to merge 11 commits into
base: master
Choose a base branch
from
Open

[improvement](be) Add disk cache for external file metadata #63376

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
9cfa41e
[refactor](be) Add file meta cache persistent interface
xylaaaaa Jul 8, 2026
283f940
[improvement](be) Add disk cache for external file metadata
xylaaaaa Jul 8, 2026
c093642
[fix](be) Fix external file meta disk cache correctness
xylaaaaa Jul 9, 2026
9b946bc
[fix](be) Guard file meta cache scanner toggles
xylaaaaa Jul 9, 2026
72e490b
[fix](be) Fix file meta cache pipeline failures
xylaaaaa Jul 10, 2026
48d7b10
[improvement](be) Add ORC v2 file meta disk cache
xylaaaaa Jul 12, 2026
677f3a2
[fix](be) Harden file metadata disk cache lifecycle
xylaaaaa Jul 12, 2026
41062d6
[test](be) Add persistent file meta cache boundary tests
xylaaaaa Jul 12, 2026
3f982d5
[fix](be) Propagate file meta cache to Iceberg v2 delete readers
xylaaaaa Jul 12, 2026
a55bd3e
[test](be) Initialize file cache dependencies in restart test
xylaaaaa Jul 12, 2026
6cfbea9
[fix](be) Harden persistent file meta cache lookup
xylaaaaa Jul 13, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions be/src/common/config.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1213,6 +1213,8 @@ DEFINE_Bool(enable_file_cache, "false");
// Both will use the directory "memory" on the disk instead of the real RAM.
DEFINE_String(file_cache_path, "[{\"path\":\"${DORIS_HOME}/file_cache\"}]");
DEFINE_Int64(file_cache_each_block_size, "1048576"); // 1MB
DEFINE_Bool(enable_external_file_meta_disk_cache, "true");
DEFINE_mInt64(external_file_meta_disk_cache_max_entry_bytes, "67108864"); // 64MB

@github-actions github-actions Bot Jun 13, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This compiled default is true, but the newly added conf/be.conf example documents # enable_external_file_meta_disk_cache = false, and the PR description frames the behavior as applying when the flag is enabled. Because ExecEnv::init_file_cache_factory() now initializes FileCacheFactory whenever this flag is true, a default BE with enable_file_cache=false will still create/use file_cache_path for persisted external footers. Please align the compiled default with the documented opt-in behavior, or update the config sample/release note/tests to make the default-on disk usage explicit.


@github-actions github-actions Bot May 26, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

enable_external_file_meta_disk_cache is declared mutable, but enabling it after startup does not actually enable the disk cache in the common default configuration. ExecEnv::init_file_cache_factory() still returns at startup when both enable_file_cache and this flag are false, so no FileCacheFactory instances are created. If an operator later sets this mutable flag to true, FileMetaCache::should_enable_for_reader() starts routing readers through the cache path, but get_block_file_cache() returns nullptr and every persistent lookup/insert silently becomes a miss/no-op. Either make this startup-only (DEFINE_Bool) or ensure the backing file-cache factory is initialized before a dynamic enable can take effect.

DEFINE_Bool(clear_file_cache, "false");
DEFINE_mBool(enable_file_cache_query_limit, "false");
Expand Down
2 changes: 2 additions & 0 deletions be/src/common/config.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1258,6 +1258,8 @@ DECLARE_Bool(enable_file_cache);
// Both will use the directory "memory" on the disk instead of the real RAM.
DECLARE_String(file_cache_path);
DECLARE_Int64(file_cache_each_block_size);
DECLARE_Bool(enable_external_file_meta_disk_cache);
DECLARE_mInt64(external_file_meta_disk_cache_max_entry_bytes);
DECLARE_Bool(clear_file_cache);
DECLARE_mBool(enable_file_cache_query_limit);
DECLARE_Int32(file_cache_enter_disk_resource_limit_mode_percent);
Expand Down
25 changes: 25 additions & 0 deletions be/src/exec/scan/file_scanner.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1346,6 +1346,12 @@ Status FileScanner::_init_parquet_reader(FileMetaCache* file_meta_cache_ptr,
std::unique_ptr<ParquetReader> parquet_reader) {
const TFileRangeDesc& range = _current_range;
Status init_status = Status::OK();
// Row-id fetch scanners read one mapped range without a split source and disable memory cache.
const bool enable_file_meta_memory_cache =
_split_source != nullptr && _should_enable_file_meta_memory_cache(file_meta_cache_ptr);
auto configure_file_meta_memory_cache = [&](GenericReader* reader) {
reader->set_enable_file_meta_memory_cache(enable_file_meta_memory_cache);
};

phmap::flat_hash_map<int, std::vector<std::shared_ptr<ColumnPredicate>>> slot_id_to_predicates =
_local_state
Expand All @@ -1371,6 +1377,7 @@ Status FileScanner::_init_parquet_reader(FileMetaCache* file_meta_cache_ptr,
[this]() -> std::shared_ptr<segment_v2::RowIdColumnIteratorV2> {
return _create_row_id_column_iterator();
});
configure_file_meta_memory_cache(iceberg_reader.get());
init_status = static_cast<GenericReader*>(iceberg_reader.get())->init_reader(&pctx);
_cur_reader = std::move(iceberg_reader);
} else if (range.__isset.table_format_params &&
Expand All @@ -1379,6 +1386,7 @@ Status FileScanner::_init_parquet_reader(FileMetaCache* file_meta_cache_ptr,
auto paimon_reader = PaimonParquetReader::create_unique(
_profile, *_params, range, _state->batch_size(), &_state->timezone_obj(), _kv_cache,
_io_ctx, _state, file_meta_cache_ptr);
configure_file_meta_memory_cache(paimon_reader.get());
init_status = static_cast<GenericReader*>(paimon_reader.get())->init_reader(&pctx);
_cur_reader = std::move(paimon_reader);
} else if (range.__isset.table_format_params &&
Expand All @@ -1387,6 +1395,7 @@ Status FileScanner::_init_parquet_reader(FileMetaCache* file_meta_cache_ptr,
auto hudi_reader = HudiParquetReader::create_unique(
_profile, *_params, range, _state->batch_size(), &_state->timezone_obj(), _io_ctx,
_state, file_meta_cache_ptr);
configure_file_meta_memory_cache(hudi_reader.get());
init_status = static_cast<GenericReader*>(hudi_reader.get())->init_reader(&pctx);
_cur_reader = std::move(hudi_reader);
} else if (range.table_format_params.table_format_type == "hive") {
Expand All @@ -1398,6 +1407,7 @@ Status FileScanner::_init_parquet_reader(FileMetaCache* file_meta_cache_ptr,
[this]() -> std::shared_ptr<segment_v2::RowIdColumnIteratorV2> {
return _create_row_id_column_iterator();
});
configure_file_meta_memory_cache(hive_reader.get());
init_status = static_cast<GenericReader*>(hive_reader.get())->init_reader(&pctx);
_cur_reader = std::move(hive_reader);
} else if (range.table_format_params.table_format_type == "tvf") {
Expand All @@ -1411,6 +1421,7 @@ Status FileScanner::_init_parquet_reader(FileMetaCache* file_meta_cache_ptr,
[this]() -> std::shared_ptr<segment_v2::RowIdColumnIteratorV2> {
return _create_row_id_column_iterator();
});
configure_file_meta_memory_cache(parquet_reader.get());
init_status = static_cast<GenericReader*>(parquet_reader.get())->init_reader(&pctx);
_cur_reader = std::move(parquet_reader);
} else if (_is_load) {
Expand All @@ -1420,6 +1431,7 @@ Status FileScanner::_init_parquet_reader(FileMetaCache* file_meta_cache_ptr,
_io_ctx, _state, file_meta_cache_ptr,
_state->query_options().enable_parquet_lazy_mat);
}
configure_file_meta_memory_cache(parquet_reader.get());
init_status = static_cast<GenericReader*>(parquet_reader.get())->init_reader(&pctx);
_cur_reader = std::move(parquet_reader);
}
Expand All @@ -1431,6 +1443,12 @@ Status FileScanner::_init_orc_reader(FileMetaCache* file_meta_cache_ptr,
std::unique_ptr<OrcReader> orc_reader) {
const TFileRangeDesc& range = _current_range;
Status init_status = Status::OK();
// Row-id fetch scanners read one mapped range without a split source and disable memory cache.
const bool enable_file_meta_memory_cache =
_split_source != nullptr && _should_enable_file_meta_memory_cache(file_meta_cache_ptr);
auto configure_file_meta_memory_cache = [&](GenericReader* reader) {
reader->set_enable_file_meta_memory_cache(enable_file_meta_memory_cache);
};

// Build unified OrcInitContext (shared by all ORC reader variants)
OrcInitContext octx;
Expand All @@ -1449,6 +1467,7 @@ Status FileScanner::_init_orc_reader(FileMetaCache* file_meta_cache_ptr,
[this]() -> std::shared_ptr<segment_v2::RowIdColumnIteratorV2> {
return _create_row_id_column_iterator();
});
configure_file_meta_memory_cache(tran_orc_reader.get());
init_status = static_cast<GenericReader*>(tran_orc_reader.get())->init_reader(&octx);

_cur_reader = std::move(tran_orc_reader);
Expand All @@ -1462,6 +1481,7 @@ Status FileScanner::_init_orc_reader(FileMetaCache* file_meta_cache_ptr,
[this]() -> std::shared_ptr<segment_v2::RowIdColumnIteratorV2> {
return _create_row_id_column_iterator();
});
configure_file_meta_memory_cache(iceberg_reader.get());
init_status = static_cast<GenericReader*>(iceberg_reader.get())->init_reader(&octx);

_cur_reader = std::move(iceberg_reader);
Expand All @@ -1471,6 +1491,7 @@ Status FileScanner::_init_orc_reader(FileMetaCache* file_meta_cache_ptr,
auto paimon_reader = PaimonOrcReader::create_unique(
_profile, _state, *_params, range, _state->batch_size(), _state->timezone(),
_kv_cache, _io_ctx, file_meta_cache_ptr);
configure_file_meta_memory_cache(paimon_reader.get());
init_status = static_cast<GenericReader*>(paimon_reader.get())->init_reader(&octx);

_cur_reader = std::move(paimon_reader);
Expand All @@ -1480,6 +1501,7 @@ Status FileScanner::_init_orc_reader(FileMetaCache* file_meta_cache_ptr,
auto hudi_reader = HudiOrcReader::create_unique(_profile, _state, *_params, range,
_state->batch_size(), _state->timezone(),
_io_ctx, file_meta_cache_ptr);
configure_file_meta_memory_cache(hudi_reader.get());
init_status = static_cast<GenericReader*>(hudi_reader.get())->init_reader(&octx);

_cur_reader = std::move(hudi_reader);
Expand All @@ -1493,6 +1515,7 @@ Status FileScanner::_init_orc_reader(FileMetaCache* file_meta_cache_ptr,
[this]() -> std::shared_ptr<segment_v2::RowIdColumnIteratorV2> {
return _create_row_id_column_iterator();
});
configure_file_meta_memory_cache(hive_reader.get());
init_status = static_cast<GenericReader*>(hive_reader.get())->init_reader(&octx);

_cur_reader = std::move(hive_reader);
Expand All @@ -1507,6 +1530,7 @@ Status FileScanner::_init_orc_reader(FileMetaCache* file_meta_cache_ptr,
[this]() -> std::shared_ptr<segment_v2::RowIdColumnIteratorV2> {
return _create_row_id_column_iterator();
});
configure_file_meta_memory_cache(orc_reader.get());
init_status = static_cast<GenericReader*>(orc_reader.get())->init_reader(&octx);
_cur_reader = std::move(orc_reader);
} else if (_is_load) {
Expand All @@ -1515,6 +1539,7 @@ Status FileScanner::_init_orc_reader(FileMetaCache* file_meta_cache_ptr,
_profile, _state, *_params, range, _state->batch_size(), _state->timezone(),
_io_ctx, file_meta_cache_ptr, _state->query_options().enable_orc_lazy_mat);
}
configure_file_meta_memory_cache(orc_reader.get());
init_status = static_cast<GenericReader*>(orc_reader.get())->init_reader(&octx);
_cur_reader = std::move(orc_reader);
}
Expand Down
14 changes: 9 additions & 5 deletions be/src/exec/scan/file_scanner.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -326,12 +326,16 @@ class FileScanner : public Scanner {
: _local_state->get_push_down_agg_type();
}

// enable the file meta cache only when
// 1. max_external_file_meta_cache_num is > 0
// 2. the file number is less than 1/3 of cache's capacibility
// Otherwise, the cache miss rate will be high
bool _should_enable_file_meta_cache() {
return ExecEnv::GetInstance()->file_meta_cache()->enabled() &&
auto* file_meta_cache = ExecEnv::GetInstance()->file_meta_cache();
return file_meta_cache != nullptr &&
(file_meta_cache->enabled() || FileMetaCache::is_persistent_cache_enabled());
}

// Enable memory file meta cache only when the file number is less than 1/3 of cache capacity.
// Otherwise, the cache miss rate will be high. Persistent cache is not gated by this rule.
bool _should_enable_file_meta_memory_cache(FileMetaCache* file_meta_cache) {
return file_meta_cache != nullptr && file_meta_cache->enabled() &&
_split_source->num_scan_ranges() < config::max_external_file_meta_cache_num / 3;
}
Comment thread
Gabriel39 marked this conversation as resolved.
};
Expand Down
15 changes: 14 additions & 1 deletion be/src/exec/scan/file_scanner_v2.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -438,6 +438,11 @@ Status FileScannerV2::_init_table_reader(const TFileRangeDesc& range) {

VExprContextSPtrs table_conjuncts;
RETURN_IF_ERROR(_build_table_conjuncts(&table_conjuncts));
FileMetaCache* file_meta_cache = nullptr;
if (_should_enable_file_meta_cache()) {
file_meta_cache = ExecEnv::GetInstance()->file_meta_cache();
DORIS_CHECK(file_meta_cache != nullptr);
}
RETURN_IF_ERROR(_table_reader->init({
.projected_columns = _projected_columns,
.conjuncts = std::move(table_conjuncts),
Expand All @@ -449,6 +454,8 @@ Status FileScannerV2::_init_table_reader(const TFileRangeDesc& range) {
.file_slot_descs = &_file_slot_descs,
.push_down_agg_type = _local_state->get_push_down_agg_type(),
.condition_cache_digest = _local_state->get_condition_cache_digest(),
.file_meta_cache = file_meta_cache,
.enable_file_meta_memory_cache = _should_enable_file_meta_memory_cache(file_meta_cache),
}));
return Status::OK();
}
Expand Down Expand Up @@ -512,7 +519,13 @@ bool FileScannerV2::_should_skip_not_found(const Status& status, bool ignore_not
}

bool FileScannerV2::_should_enable_file_meta_cache() const {
return ExecEnv::GetInstance()->file_meta_cache()->enabled() &&
auto* file_meta_cache = ExecEnv::GetInstance()->file_meta_cache();
return file_meta_cache != nullptr &&
(file_meta_cache->enabled() || FileMetaCache::is_persistent_cache_enabled());
}

bool FileScannerV2::_should_enable_file_meta_memory_cache(FileMetaCache* file_meta_cache) const {
return file_meta_cache != nullptr && file_meta_cache->enabled() &&
_split_source->num_scan_ranges() < config::max_external_file_meta_cache_num / 3;
}

Expand Down
1 change: 1 addition & 0 deletions be/src/exec/scan/file_scanner_v2.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -120,6 +120,7 @@ class FileScannerV2 final : public Scanner {
std::map<std::string, Field> partition_values);
static bool _should_skip_not_found(const Status& status, bool ignore_not_found);
bool _should_enable_file_meta_cache() const;
bool _should_enable_file_meta_memory_cache(FileMetaCache* file_meta_cache) const;
std::optional<format::GlobalRowIdContext> _create_global_rowid_context(
const TFileRangeDesc& range) const;
Status _generate_partition_values(const TFileRangeDesc& range,
Expand Down
3 changes: 3 additions & 0 deletions be/src/format/generic_reader.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -258,6 +258,8 @@ class GenericReader : public ProfileCollector {
// Pass condition cache context to the reader for HIT/MISS tracking.
virtual void set_condition_cache_context(std::shared_ptr<ConditionCacheContext> ctx) {}

void set_enable_file_meta_memory_cache(bool enable) { _enable_file_meta_memory_cache = enable; }

// Returns true if this reader can produce an accurate total row count from metadata
// without reading actual data. Used to determine if CountReader decorator can be applied.
// Only ORC and Parquet readers support this (via file footer metadata).
Expand All @@ -279,6 +281,7 @@ class GenericReader : public ProfileCollector {
// Cache to save some common part such as file footer.
// Maybe null if not used
FileMetaCache* _meta_cache = nullptr;
bool _enable_file_meta_memory_cache = true;

// ---- Column descriptors (set by init_reader, owned by FileScanner) ----
const std::vector<ColumnDescriptor>* _column_descs = nullptr;
Expand Down
Loading
Loading