Skip to content

Commit dcc6924

Browse files
suxiaogang223suxiaogang223
committed
docs: align unified external meta cache docs
1 parent e5e4ae0 commit dcc6924

16 files changed

Lines changed: 373 additions & 201 deletions

File tree

‎docs/admin-manual/system-tables/information_schema/catalog_meta_cache_statistics.md‎

Lines changed: 37 additions & 35 deletions
Original file line numberDiff line numberDiff line change
@@ -18,46 +18,48 @@ View the metadata cache information of the External Catalog in the currently con
1818

1919
## Table Information
2020

21-
| Column Name | Type | Description |
22-
| ------------ | ---- | ----------------------- |
23-
| CATALOG_NAME | text | The name of the Catalog |
24-
| CACHE_NAME | text | The name of the cache |
25-
| METRIC_NAME | text | The name of the metric |
26-
| METRIC_VALUE | text | The value of the metric |
21+
One row represents one cache entry on one FE for one external catalog.
22+
23+
| Column Name | Type | Description |
24+
| ------------ | ---- | ----------- |
25+
| FE_HOST | text | FE host that reports the stats |
26+
| CATALOG_NAME | text | Catalog name |
27+
| ENGINE_NAME | text | Meta cache engine name, such as `hive`, `iceberg`, `paimon` |
28+
| ENTRY_NAME | text | Cache entry name inside the engine, such as `schema`, `file`, `manifest` |
29+
| EFFECTIVE_ENABLED | boolean | Whether the cache is effectively enabled after evaluating `enable` / `ttl-second` / `capacity` |
30+
| CONFIG_ENABLED | boolean | Raw `enable` flag from the cache config |
31+
| AUTO_REFRESH | boolean | Whether async refresh-after-write is enabled for this entry |
32+
| TTL_SECOND | bigint | TTL in seconds. `0` means disabled; `-1` means no expiration |
33+
| CAPACITY | bigint | Max entry count |
34+
| ESTIMATED_SIZE | bigint | Estimated current cache size |
35+
| REQUEST_COUNT | bigint | Total requests |
36+
| HIT_COUNT | bigint | Cache hits |
37+
| MISS_COUNT | bigint | Cache misses |
38+
| HIT_RATE | double | Hit rate |
39+
| LOAD_SUCCESS_COUNT | bigint | Successful loads |
40+
| LOAD_FAILURE_COUNT | bigint | Failed loads |
41+
| TOTAL_LOAD_TIME_MS | bigint | Total load time in milliseconds |
42+
| AVG_LOAD_PENALTY_MS | double | Average load time in milliseconds |
43+
| EVICTION_COUNT | bigint | Evicted entries |
44+
| INVALIDATE_COUNT | bigint | Explicit invalidations |
45+
| LAST_LOAD_SUCCESS_TIME | text | Last successful load time |
46+
| LAST_LOAD_FAILURE_TIME | text | Last failed load time |
47+
| LAST_ERROR | text | Latest load error message |
2748

2849

2950
## Usage Example
3051

31-
```text
32-
+----------------------+-----------------------------+----------------------+---------------------+
33-
| CATALOG_NAME | CACHE_NAME | METRIC_NAME | METRIC_VALUE |
34-
+----------------------+-----------------------------+----------------------+---------------------+
35-
| hive_iceberg_minio | iceberg_table_cache | eviction_count | 0 |
36-
| hive_iceberg_minio | iceberg_table_cache | hit_ratio | 0.8235294117647058 |
37-
| hive_iceberg_minio | iceberg_table_cache | average_load_penalty | 5.480102048333334E8 |
38-
| hive_iceberg_minio | iceberg_table_cache | estimated_size | 6 |
39-
| hive_iceberg_minio | iceberg_table_cache | hit_count | 28 |
40-
| hive_iceberg_minio | iceberg_table_cache | read_count | 34 |
41-
| hive_iceberg_minio | iceberg_snapshot_list_cache | eviction_count | 0 |
42-
| hive_iceberg_minio | iceberg_snapshot_list_cache | hit_ratio | 1.0 |
43-
| hive_iceberg_minio | iceberg_snapshot_list_cache | average_load_penalty | 0.0 |
44-
| hive_iceberg_minio | iceberg_snapshot_list_cache | estimated_size | 0 |
45-
| hive_iceberg_minio | iceberg_snapshot_list_cache | hit_count | 0 |
46-
| hive_iceberg_minio | iceberg_snapshot_list_cache | read_count | 0 |
47-
| hive_iceberg_minio | iceberg_snapshot_cache | eviction_count | 0 |
48-
| hive_iceberg_minio | iceberg_snapshot_cache | hit_ratio | 0.45454545454545453 |
49-
| hive_iceberg_minio | iceberg_snapshot_cache | average_load_penalty | 5.604907246666666E8 |
50-
| hive_iceberg_minio | iceberg_snapshot_cache | estimated_size | 6 |
51-
| hive_iceberg_minio | iceberg_snapshot_cache | hit_count | 5 |
52-
| hive_iceberg_minio | iceberg_snapshot_cache | read_count | 11 |
52+
```sql
53+
SELECT catalog_name, engine_name, entry_name,
54+
effective_enabled, ttl_second, capacity,
55+
estimated_size, hit_rate, last_error
56+
FROM information_schema.catalog_meta_cache_statistics
57+
ORDER BY catalog_name, engine_name, entry_name;
5358
```
5459

55-
The METRIC_NAME column contains the following Caffeine cache performance metrics:
56-
- eviction_count: The number of entries that have been evicted from the cache
57-
- hit_ratio: The ratio of cache requests which were hits (ranges from 0.0 to 1.0)
58-
- average_load_penalty: The average time spent loading new values (in nanoseconds)
59-
- estimated_size: The approximate number of entries in the cache
60-
- hit_count: The number of times cache lookup methods have returned a cached value
61-
- read_count: The total number of times cache lookup methods have been called
60+
Typical usage:
6261

62+
- Use `ENGINE_NAME` + `ENTRY_NAME` to identify one logical cache entry.
63+
- Use `EFFECTIVE_ENABLED`, `TTL_SECOND`, and `CAPACITY` to confirm the applied cache policy.
64+
- Use `HIT_RATE`, `ESTIMATED_SIZE`, `LOAD_FAILURE_COUNT`, and `LAST_ERROR` to diagnose behavior.
6365

‎docs/lakehouse/catalogs/hive-catalog.mdx‎

Lines changed: 22 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -79,19 +79,26 @@ CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES (
7979
## Metadata Cache (4.1.x+) {#meta-cache-unified}
8080

8181
Starting from Doris 4.1.x, Hive Catalog metadata caches are configured with the unified `meta.cache.*` properties.
82-
This section focuses on **how to use** and **how to observe** the Hive-related cache modules.
82+
This section covers configuration and observability for Hive-related cache modules.
8383

8484
For the unified property semantics, see: [Unified External Meta Cache (4.1.x+)](../meta-cache/unified-meta-cache.md).
8585

8686
### Cache Modules {#meta-cache-unified-modules}
8787

8888
| Module | Property key prefix | Cached content (typical) |
8989
|---|---|---|
90-
| `partition-values` | `meta.cache.hive.partition-values.` | Partition values/names list used by partition pruning and partition enumeration. |
90+
| `schema` | `meta.cache.hive.schema.` | Schema cache entry for table schema loading. |
91+
| `partition_values` | `meta.cache.hive.partition_values.` | Partition values/names list used by partition pruning and partition enumeration. |
9192
| `partition` | `meta.cache.hive.partition.` | Partition properties (location, input format, storage descriptor, etc.). |
9293
| `file` | `meta.cache.hive.file.` | File listing under partition/table paths (reduces remote LIST overhead). |
9394

94-
Example (disable file listing cache for freshness):
95+
Notes:
96+
97+
- Property keys use the module names shown above. The same names appear as `ENTRY_NAME` in `information_schema.catalog_meta_cache_statistics`.
98+
- Changing legacy properties such as `file.meta.cache.ttl-second` and `partition.cache.ttl-second` can trigger Hive cache rebuild behavior.
99+
- Changing unified `meta.cache.hive.*` properties on an already-initialized catalog does not fully hot-reload existing Hive cache entries in current releases. To guarantee that a new cache spec is applied, recreate the catalog or restart FE.
100+
101+
Example:
95102

96103
```sql
97104
ALTER CATALOG hive_ctl SET PROPERTIES (
@@ -104,22 +111,25 @@ ALTER CATALOG hive_ctl SET PROPERTIES (
104111
Hive cache metrics are available in `information_schema.catalog_meta_cache_statistics`.
105112
For the table definition and metric meanings, see: [catalog_meta_cache_statistics](../../admin-manual/system-tables/information_schema/catalog_meta_cache_statistics.md).
106113

107-
The `cache_name` values for Hive modules are:
114+
Common Hive entries:
108115

109-
| Module | cache_name |
116+
| Entry | Meaning |
110117
|---|---|
111-
| `partition-values` | `hive_partition_values_cache` |
112-
| `partition` | `hive_partition_cache` |
113-
| `file` | `hive_file_cache` |
118+
| `schema` | Schema cache entry |
119+
| `partition_values` | Partition names / values cache entry |
120+
| `partition` | Partition property cache entry |
121+
| `file` | File listing cache entry |
114122

115-
Example query (filter one catalog and Hive caches):
123+
Example query:
116124

117125
```sql
118-
SELECT *
126+
SELECT catalog_name, engine_name, entry_name,
127+
effective_enabled, ttl_second, capacity,
128+
estimated_size, hit_rate, load_failure_count, last_error
119129
FROM information_schema.catalog_meta_cache_statistics
120130
WHERE catalog_name = 'hive_ctl'
121-
AND cache_name LIKE 'hive_%'
122-
ORDER BY cache_name, metric_name;
131+
AND engine_name = 'hive'
132+
ORDER BY entry_name;
123133
```
124134

125135
### Supported Hive Versions

‎docs/lakehouse/catalogs/hudi-catalog.md‎

Lines changed: 22 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -54,23 +54,29 @@ CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES (
5454
## Metadata Cache (4.1.x+) {#meta-cache-unified}
5555

5656
Starting from Doris 4.1.x, Hudi-related metadata caches are configured with the unified `meta.cache.*` properties.
57-
This section focuses on **how to use** and **how to observe** the Hudi cache modules.
57+
This section covers configuration and observability for Hudi-related cache modules.
5858

5959
For the unified property semantics, see: [Unified External Meta Cache (4.1.x+)](../meta-cache/unified-meta-cache.md).
6060

6161
### Cache Modules {#meta-cache-unified-modules}
6262

6363
| Module | Property key prefix | Cached content (typical) |
6464
|---|---|---|
65+
| `schema` | `meta.cache.hudi.schema.` | Schema cache entry for table schema loading. |
6566
| `partition` | `meta.cache.hudi.partition.` | Hudi partition-related metadata (used by partition discovery/pruning). |
66-
| `fs-view` | `meta.cache.hudi.fs-view.` | Hudi filesystem view related metadata. |
67-
| `meta-client` | `meta.cache.hudi.meta-client.` | Hudi meta client related metadata. |
67+
| `fs_view` | `meta.cache.hudi.fs_view.` | Hudi filesystem view related metadata. |
68+
| `meta_client` | `meta.cache.hudi.meta_client.` | Hudi meta client related metadata. |
6869

69-
Example (reduce cache footprint by lowering capacity):
70+
Notes:
71+
72+
- Property keys use the module names shown above. The same names appear as `ENTRY_NAME` in `information_schema.catalog_meta_cache_statistics`.
73+
- When Hudi tables are accessed through an HMS catalog, configure `meta.cache.hudi.*` on that HMS catalog.
74+
75+
Example:
7076

7177
```sql
7278
ALTER CATALOG hudi_ctl SET PROPERTIES (
73-
"meta.cache.hudi.partition.capacity" = "2000"
79+
"meta.cache.hudi.fs_view.capacity" = "2000"
7480
);
7581
```
7682

@@ -79,22 +85,25 @@ ALTER CATALOG hudi_ctl SET PROPERTIES (
7985
Hudi cache metrics are available in `information_schema.catalog_meta_cache_statistics`.
8086
For the table definition and metric meanings, see: [catalog_meta_cache_statistics](../../admin-manual/system-tables/information_schema/catalog_meta_cache_statistics.md).
8187

82-
The `cache_name` values for Hudi modules are:
88+
Common Hudi entries:
8389

84-
| Module | cache_name |
90+
| Entry | Meaning |
8591
|---|---|
86-
| `partition` | `hudi_partition_cache` |
87-
| `fs-view` | `hudi_fs_view_cache` |
88-
| `meta-client` | `hudi_meta_client_cache` |
92+
| `schema` | Schema cache entry |
93+
| `partition` | Partition metadata cache entry |
94+
| `fs_view` | File system view cache entry |
95+
| `meta_client` | Meta client cache entry |
8996

9097
Example query:
9198

9299
```sql
93-
SELECT *
100+
SELECT catalog_name, engine_name, entry_name,
101+
effective_enabled, ttl_second, capacity,
102+
estimated_size, hit_rate, load_failure_count, last_error
94103
FROM information_schema.catalog_meta_cache_statistics
95104
WHERE catalog_name = 'hudi_ctl'
96-
AND cache_name LIKE 'hudi_%'
97-
ORDER BY cache_name, metric_name;
105+
AND engine_name = 'hudi'
106+
ORDER BY entry_name;
98107
```
99108

100109
### Supported Hudi Versions

‎docs/lakehouse/catalogs/iceberg-catalog.mdx‎

Lines changed: 23 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -88,21 +88,31 @@ CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES (
8888
## Metadata Cache (4.1.x+) {#meta-cache-unified}
8989

9090
Starting from Doris 4.1.x, Iceberg Catalog metadata caches are configured with the unified `meta.cache.*` properties.
91-
This section focuses on **how to use** and **how to observe** the Iceberg-related cache modules.
91+
This section covers configuration and observability for Iceberg-related cache modules.
9292

9393
For the unified property semantics, see: [Unified External Meta Cache (4.1.x+)](../meta-cache/unified-meta-cache.md).
9494

9595
### Cache Modules {#meta-cache-unified-modules}
9696

9797
| Module | Property key prefix | Cached content (typical) |
9898
|---|---|---|
99+
| `schema` | `meta.cache.iceberg.schema.` | Schema cache entry for table schema loading. |
99100
| `table` | `meta.cache.iceberg.table.` | Iceberg table metadata object (reduces catalog/metastore round trips). |
101+
| `view` | `meta.cache.iceberg.view.` | Iceberg view metadata object. |
100102
| `manifest` | `meta.cache.iceberg.manifest.` | Manifest-related metadata (reduces repeated manifest access overhead). |
101103

102-
Example (shorter TTL for manifest to prioritize freshness):
104+
Notes:
105+
106+
- Property keys use the module names shown above. The same names appear as `ENTRY_NAME` in `information_schema.catalog_meta_cache_statistics`.
107+
- `manifest` is disabled by default in the current implementation. Enable it explicitly before tuning TTL/capacity.
108+
- `view` entries are only populated when Doris accesses Iceberg views.
109+
- `ALTER CATALOG ... SET PROPERTIES` updates are applied through the unified hot-reload path.
110+
111+
Example:
103112

104113
```sql
105114
ALTER CATALOG iceberg_ctl SET PROPERTIES (
115+
"meta.cache.iceberg.manifest.enable" = "true",
106116
"meta.cache.iceberg.manifest.ttl-second" = "600"
107117
);
108118
```
@@ -112,21 +122,25 @@ ALTER CATALOG iceberg_ctl SET PROPERTIES (
112122
Iceberg cache metrics are available in `information_schema.catalog_meta_cache_statistics`.
113123
For the table definition and metric meanings, see: [catalog_meta_cache_statistics](../../admin-manual/system-tables/information_schema/catalog_meta_cache_statistics.md).
114124

115-
The `cache_name` values for Iceberg modules are:
125+
Common Iceberg entries:
116126

117-
| Module | cache_name |
127+
| Entry | Meaning |
118128
|---|---|
119-
| `table` | `iceberg_table_cache` |
120-
| `manifest` | `iceberg_manifest_cache` |
129+
| `schema` | Schema cache entry |
130+
| `table` | Table metadata cache entry |
131+
| `view` | View metadata cache entry |
132+
| `manifest` | Manifest payload cache entry |
121133

122134
Example query:
123135

124136
```sql
125-
SELECT *
137+
SELECT catalog_name, engine_name, entry_name,
138+
effective_enabled, ttl_second, capacity,
139+
estimated_size, hit_rate, load_failure_count, last_error
126140
FROM information_schema.catalog_meta_cache_statistics
127141
WHERE catalog_name = 'iceberg_ctl'
128-
AND cache_name LIKE 'iceberg_%'
129-
ORDER BY cache_name, metric_name;
142+
AND engine_name = 'iceberg'
143+
ORDER BY entry_name;
130144
```
131145

132146
### Supported Iceberg Versions

‎docs/lakehouse/catalogs/maxcompute-catalog.md‎

Lines changed: 17 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -71,44 +71,46 @@ CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES (
7171
## Metadata Cache (4.1.x+) {#meta-cache-unified}
7272

7373
Starting from Doris 4.1.x, MaxCompute Catalog metadata caches are configured with the unified `meta.cache.*` properties.
74-
This section focuses on **how to use** and **how to observe** the MaxCompute-related cache module.
74+
This section covers configuration and observability for MaxCompute-related cache modules.
7575

7676
For the unified property semantics, see: [Unified External Meta Cache (4.1.x+)](../meta-cache/unified-meta-cache.md).
7777

7878
### Cache Modules {#meta-cache-unified-modules}
7979

8080
| Module | Property key prefix | Cached content (typical) |
8181
|---|---|---|
82-
| `partition-values` | `meta.cache.maxcompute.partition-values.` | Partition values list (reduces repeated remote listing overhead). |
82+
| `schema` | `meta.cache.maxcompute.schema.` | Schema cache entry for table schema loading. |
83+
| `partition_values` | `meta.cache.maxcompute.partition_values.` | Partition values cache entry used for partition pruning and partition enumeration. |
8384

84-
Example:
85+
Notes:
8586

86-
```sql
87-
ALTER CATALOG mc_ctl SET PROPERTIES (
88-
"meta.cache.maxcompute.partition-values.ttl-second" = "3600",
89-
"meta.cache.maxcompute.partition-values.capacity" = "5000"
90-
);
91-
```
87+
- Property keys use the module names shown above. The same names appear as `ENTRY_NAME` in `information_schema.catalog_meta_cache_statistics`.
88+
- `partition_values` is configured through `meta.cache.maxcompute.partition_values.*`.
89+
- The stats table exposes `partition_values` and `schema` as the two MaxCompute entries.
90+
- There is no dedicated MaxCompute catalog-level hot-reload hook for `meta.cache.maxcompute.*`.
9291

9392
### Observability {#meta-cache-unified-observability}
9493

9594
MaxCompute cache metrics are available in `information_schema.catalog_meta_cache_statistics`.
9695
For the table definition and metric meanings, see: [catalog_meta_cache_statistics](../../admin-manual/system-tables/information_schema/catalog_meta_cache_statistics.md).
9796

98-
The `cache_name` value for MaxCompute module is:
97+
Common MaxCompute entries:
9998

100-
| Module | cache_name |
99+
| Entry | Meaning |
101100
|---|---|
102-
| `partition-values` | `maxcompute_partition_values_cache` |
101+
| `schema` | Schema cache entry |
102+
| `partition_values` | Partition values cache entry |
103103

104104
Example query:
105105

106106
```sql
107-
SELECT *
107+
SELECT catalog_name, engine_name, entry_name,
108+
effective_enabled, ttl_second, capacity,
109+
estimated_size, hit_rate, load_failure_count, last_error
108110
FROM information_schema.catalog_meta_cache_statistics
109111
WHERE catalog_name = 'mc_ctl'
110-
AND cache_name LIKE 'maxcompute_%'
111-
ORDER BY cache_name, metric_name;
112+
AND engine_name = 'maxcompute'
113+
ORDER BY entry_name;
112114
```
113115

114116
### Supported MaxCompute Versions

0 commit comments

Comments
 (0)