[doc](lakehouse) Add external metadata cache memory management for 4.1.4 - #4061
Draft
CalvinKirs wants to merge 1 commit into
Draft
[doc](lakehouse) Add external metadata cache memory management for 4.1.4#4061CalvinKirs wants to merge 1 commit into
CalvinKirs wants to merge 1 commit into
Conversation
CalvinKirs
force-pushed
the
agent/external-meta-cache-memory-doc-4.1.5
branch
from
August 15, 2026 07:42
2125f05 to
03960ee
Compare
yiguolei
pushed a commit
to apache/doris
that referenced
this pull request
Aug 25, 2026
…overnance (#66717) ## DRAFT Docs https://github.com/CalvinKirs/doris-website/blob/2125f053594b821a6ab7556f035b9cb1e5b43a0f/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/lakehouse/external-meta-cache-memory-management.md apache/doris-website#4061 (comment) ## Summary Add retained-memory governance for selected external metadata caches. Existing count-based capacity remains the default. Weighted admission is enabled only for an estimator-backed entry when at least one applicable global, catalog, or entry memory limit is configured. ## Why - Entry count does not bound FE heap when cached metadata sizes are highly uneven. - Small-sample estimation can miss a large tail element. - Generic reflective traversal is brittle across Iceberg/Paimon upgrades and JVM module boundaries. - Shared infrastructure such as FileIO, Catalog, Hadoop configuration, clients, and executors must not be charged to every cache item. - Memory accounting must not become the dominant metadata-loading cost. ## Managed scope - Hive: `partition_values`. - Iceberg: `table`, `snapshot`, and `manifest` (`manifest` remains disabled by default). - Paimon: `snapshot`. Other external metadata entries continue to use their existing count-based behavior. ## Accounting and ownership strategy 1. Supported cache values expose explicit retained-size counters and conservative formulas; no private-field reflection or generic object-graph traversal is used. 2. Variable payload is counted while the loader is already constructing owned collections. Publication stores the completed estimate, so Caffeine weighing and later cache hits are O(1). 3. Shared infrastructure is an ownership boundary and is not charged to each item. 4. Weighted Caffeine caches use soft values. Reservation records retain only key/generation/weight ownership, not a strong reference to the value, so GC collection can release the matching reservation. 5. Admission/replacement and reservation ownership changes are serialized atomically. Removal releases only the matching generation and cannot release a concurrently published replacement. 6. Local cold entries are evicted before an admission is rejected. There is no cross-catalog global LRU. Iceberg table/snapshot cache values use a detached, non-growing metadata generation. Historical refs/snapshots/statistics are not retained by the cache entry; a statement that needs them reads the exact pinned metadata file into a query-local table under the catalog authenticator. The statement keeps one generation even if the cache concurrently refreshes. A stale unbound cache generation is invalidated and retried once; an already-bound statement fails instead of silently switching generations. Snapshot identity includes `metadataFileLocation + snapshotId + schemaId + defaultSpecId`. Paimon partition payload bytes are accumulated in the existing partition-construction loop, including every retained typed value and display name. This avoids sampling misses without a second full traversal. ## Limit behavior - A value larger than the effective entry limit is returned to the current request but is not cached. - If local eviction still cannot satisfy global/catalog/entry admission, the loaded value is returned but is not cached; normal budget rejection does not fail the query. - Incomplete or failed preparation also fails cache admission closed rather than contributing zero bytes. - A rejected refresh does not publish known-stale metadata. - Limits govern retained cache memory after construction. They are not a pre-load heap reservation, so a remote load failure or OOM while building one exceptionally large value can still fail before admission. ## Configuration - FE total: `external_meta_cache_max_weight=10GB` or `20%`; `0` disables the FE-global quota. - Catalog total: `meta.cache.max-weight=4GB`. - Optional entry override: `meta.cache.<engine>.<entry>.max-weight=1GB`. Not every entry needs an explicit limit. Estimator-backed entries inherit the nearest configured parent. Catalog/entry limits also work when the FE-global limit is disabled. The hierarchy is validated as `entry <= catalog <= global` when the corresponding parents exist. Unknown engines, entries, options, aliases, and max-weight on entries without an estimator are rejected during catalog validation. ## Optimizer and query-path impact No optimizer rule, literal representation, partition-item implementation, or system-table exposure is added. The only scan-node edit stores an existing `Optional` result once before use; it does not change scan planning semantics. ## Validation - Focused Maven reactor regression: 142 tests, 0 failures, 0 errors. - Checkstyle: 0 violations; `git diff --check` passes. - Earlier feature-branch integration smoke: 6 real catalogs queried successfully; observed managed weight stayed at `520964 <= 524288`; budget rejection did not fail queries; no incomplete estimate, accounting underflow, deadlock, or OOM was observed. The latest source behavior is covered by the focused unit regression above. ## Performance results In-repo benchmark harness, Java 17, `-Xms1g -Xmx4g`, 500 ms warmup and 3 x 500 ms measurement. Results are per operation. | Case | Baseline | With retained counter | Added cost | | --- | ---: | ---: | ---: | | Hive 100k uniform partitions | 191.591 ms | 205.837 ms | +7.4% | | Hive 100k tail-skew partitions | 209.541 ms | 191.196 ms | within run variance | | Paimon 1k uniform partitions | 185.353 us | 199.803 us | +7.8% | | Paimon 1k tail-skew partitions | 179.675 us | 202.832 us | +12.9% | | Paimon 10k uniform partitions | 1825.430 us | 2052.980 us | +12.5% | | Paimon 10k tail-skew partitions | 1959.609 us | 2134.038 us | +8.9% | | Iceberg manifest, 100 files x 100 metric columns | 462.898 us | 692.103 us | +49.5% | | Iceberg manifest, 10k files x 100 metric columns | 48185.622 us | 72071.263 us | +49.6% | | Iceberg manifest, 100 files x 1000 metric columns | 5620.316 us | 7772.058 us | +38.3% | | Iceberg manifest, 10k files x 1000 metric columns | 808987.032 us | 1049782.729 us | +29.8% | The Iceberg comparison includes `DataFile.copy()` in both paths, matching the production manifest reader. Even in the dense-metrics stress cases, copying/parsing remains the larger component than the incremental counter. Iceberg table publication is 4.401 us (10 fields) / 9.991 us (100 fields); 1k versus 10k retained snapshot history is 2.931 us / 3.006 us, showing no history-length traversal. Prepared weight lookup is approximately 30-40 ns for Iceberg/Paimon.
CalvinKirs
force-pushed
the
agent/external-meta-cache-memory-doc-4.1.5
branch
from
August 25, 2026 04:10
03960ee to
452f13f
Compare
CalvinKirs
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
partition_values; Icebergtable,snapshot, andmanifest; Paimontableandsnapshot.catalog_meta_cache_statisticsin both languages to match the 33-column FE schema, including weight usage, eviction, and rejection fields.Runtime behavior covered
CachingCatalogdefaults when Doris memory governance is active, including explicit-property precedence.This documents the user-facing behavior introduced by apache/doris#66717.
Checks
apache/doris-website:master.git diff --check.SchemaTable.java.A full Docusaurus build was not run because dependencies are not installed in the clean worktree.