Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@
"description": "Doris uses Jemalloc as the general memory allocator by default. The memory occupied by Jemalloc itself includes Cache and Metadata."
}
---

Check warning on line 8 in docs/admin-manual/trouble-shooting/memory-management/memory-analysis/jemalloc-memory-analysis.md

View workflow job for this annotation

GitHub Actions / Build Check

markdown-code-fence-language

Code fence should declare a language. Owner%3A @apache/doris-website-maintainers
Doris uses Jemalloc as the general memory allocator by default. The memory occupied by Jemalloc itself includes Cache and Metadata. Cache includes Thread Cache and Dirty Page. You can view the original profile of the memory allocator in real time at http://{be_host}:{be_web_server_port}/memz.
Doris uses Jemalloc as the general memory allocator by default. The memory occupied by Jemalloc itself includes Cache and Metadata. Cache includes Thread Cache and Dirty Page. You can view the original profile of the memory allocator in real time in the **Jemalloc Profiles** section under **Memory Info** on Doris BE's Process Profile Web page http://{be_host}:{be_web_server_port}/profile.

## Jemalloc Cache Memory Analysis

Expand All @@ -20,13 +20,13 @@

During the running of the BE process, Jemalloc Cache consists of two parts.

- Thread Cache, cache a specified number of Pages in Thread Cache, refer to [Jemalloc opt.tcache](https://jemalloc.net/jemalloc.3.html#opt.tcache).

Check notice on line 23 in docs/admin-manual/trouble-shooting/memory-management/memory-analysis/jemalloc-memory-analysis.md

View workflow job for this annotation

GitHub Actions / Build Check

link-external-report-only

External link is report-only and was not fetched%3A https%3A//jemalloc.net/jemalloc.3.html#opt.tcache. Owner%3A @apache/doris-website-maintainers

- Dirty Page, all memory Pages that can be reused in Arena.

### Jemalloc Cache View Method

View Doris BE's Web page `http://{be_host}:{be_web_server_port}/memz` (webserver_port defaults to 8040) to obtain Jemalloc Profile, and interpret the use of Jemalloc Cache based on several sets of key information.
View the **Jemalloc Profiles** section under **Memory Info** on Doris BE's Process Profile Web page `http://{be_host}:{be_web_server_port}/profile` (webserver_port defaults to 8040) to obtain the Jemalloc Profile, and interpret the use of Jemalloc Cache based on several sets of key information.

- `tcache_bytes` in Jemalloc Profile is the total number of bytes of Jemalloc Thread Cache. If the `tcache_bytes` value is large, it means that the memory used by Jemalloc Thread Cache is too large.

Expand All @@ -38,7 +38,7 @@

Consider reducing `lg_tcache_max` in `JEMALLOC_CONF` in `be.conf`. `lg_tcache_max` is the upper limit of the byte size of the page allowed to be cached. The default value is 15, that is, 32 KB (2^15). Pages exceeding this size will not be cached in the Thread Cache. `lg_tcache_max` corresponds to `Maximum thread-cached size class` in the Jemalloc Profile.

> Before Doris 2.1, the default value of `lg_tcache_max` in `JEMALLOC_CONF` in `be.conf` is 20, which will cause the Jemalloc Cache to be too large in some scenarios. After Doris 2.1, it has been changed back to the default value of Jemalloc 15.

Check warning on line 41 in docs/admin-manual/trouble-shooting/memory-management/memory-analysis/jemalloc-memory-analysis.md

View workflow job for this annotation

GitHub Actions / Build Check

markdown-code-fence-language

Code fence should declare a language. Owner%3A @apache/doris-website-maintainers

This is usually because the query or load in the BE process is applying for a large number of memory pages of large size classes, or after executing a large memory query or load, a large number of memory pages of large size classes are cached in the Thread Cache. There are two times to clean up the Thread Cache. One is to recycle the memory blocks that have not been used for a long time when the memory application and release reach a certain number of times; the other is to recycle all pages when the thread exits. At this time, there is a Bad Case. If the thread has not executed new queries or loads in the future, it will no longer allocate memory and fall into a so-called `idle` state. Users expect that the memory can be released after the query is completed, but in fact, in this scenario, if the thread does not exit, the Thread Cache will not be cleaned.

Expand All @@ -64,11 +64,11 @@
114688 14 1 102400 1 106496 15 1642496 17 1851392
```

Reduce `dirty_decay_ms` of `JEMALLOC_CONF` in `be.conf` to 2000 ms or less. The default `dirty_decay_ms` in `be.conf` is 5000 ms. Jemalloc will release dirty pages according to a smooth gradient curve within the time specified by `dirty_decay_ms`. For reference, [Jemalloc opt.dirty_decay_ms](https://jemalloc.net/jemalloc.3.html#opt.dirty_decay_ms). When the BE process has insufficient available memory and triggers Minor GC or Full GC, it will actively release all dirty pages according to a certain strategy.

Check notice on line 67 in docs/admin-manual/trouble-shooting/memory-management/memory-analysis/jemalloc-memory-analysis.md

View workflow job for this annotation

GitHub Actions / Build Check

link-external-report-only

External link is report-only and was not fetched%3A https%3A//jemalloc.net/jemalloc.3.html#opt.dirty_decay_ms. Owner%3A @apache/doris-website-maintainers

> Before Doris 2.1, the default value of `dirty_decay_ms` in `JEMALLOC_CONF` in `be.conf` is 15000, which will cause the Jemalloc Cache to be too large in some scenarios. After Doris 2.1, the default value is 5000.

Check warning on line 70 in docs/admin-manual/trouble-shooting/memory-management/memory-analysis/jemalloc-memory-analysis.md

View workflow job for this annotation

GitHub Actions / Build Check

markdown-code-fence-language

Code fence should declare a language. Owner%3A @apache/doris-website-maintainers
`extents` in Jemalloc Profile contains the statistical values ​​of buckets of different page sizes in all Jemalloc `arena`, where `ndirty` is the number of dirty pages and `dirty` is the total memory of dirty pages. Refer to `stats.arenas.<i>.extents.<j>.{extent_type}_bytes` in [Jemalloc](https://jemalloc.net/jemalloc.3.html) and add up the `dirty` of all Page Sizes to get the memory byte size of the Dirty Page in Jemalloc.

Check notice on line 71 in docs/admin-manual/trouble-shooting/memory-management/memory-analysis/jemalloc-memory-analysis.md

View workflow job for this annotation

GitHub Actions / Build Check

link-external-report-only

External link is report-only and was not fetched%3A https%3A//jemalloc.net/jemalloc.3.html. Owner%3A @apache/doris-website-maintainers

## Jemalloc Metadata Memory Analysis

Expand All @@ -82,7 +82,7 @@

### How to view Jemalloc Metadata

You can get the Jemalloc Profile by viewing the Doris BE web page `http://{be_host}:{be_web_server_port}/memz` (webserver_port defaults to 8040). Find the overall memory statistics of Jemalloc in the Jemalloc Profile as follows, where `metadata` is the memory size of Jemalloc Metadata.
You can get the Jemalloc Profile from the **Jemalloc Profiles** section under **Memory Info** on the Doris BE Process Profile web page `http://{be_host}:{be_web_server_port}/profile` (webserver_port defaults to 8040). Find the overall memory statistics of Jemalloc in the Jemalloc Profile as follows, where `metadata` is the memory size of Jemalloc Metadata.

`Allocated: 2401232080, active: 2526302208, metadata: 535979296 (n_thp 221), resident: 2995621888, mapped: 3221979136, retained: 131542581248`

Expand All @@ -90,9 +90,9 @@

- `active` The total number of bytes of all pages allocated by Jemalloc for the BE process, which is a multiple of Page Size and is usually greater than or equal to `Allocated`.

- `metadata` The total number of bytes of Jemalloc metadata, which is related to the number of allocated and cached pages, memory fragmentation and other factors. Refer to the document [Jemalloc stats.metadata](https://jemalloc.net/jemalloc.3.html#stats.metadata)

Check notice on line 93 in docs/admin-manual/trouble-shooting/memory-management/memory-analysis/jemalloc-memory-analysis.md

View workflow job for this annotation

GitHub Actions / Build Check

link-external-report-only

External link is report-only and was not fetched%3A https%3A//jemalloc.net/jemalloc.3.html#stats.metadata. Owner%3A @apache/doris-website-maintainers

- `retained` The size of the virtual memory mapping retained by Jemalloc, which is not returned to the operating system through munmap or similar methods, and is not strongly associated with physical memory. Reference document [Jemalloc stats.retained](https://jemalloc.net/jemalloc.3.html#stats.retained)

Check notice on line 95 in docs/admin-manual/trouble-shooting/memory-management/memory-analysis/jemalloc-memory-analysis.md

View workflow job for this annotation

GitHub Actions / Build Check

link-external-report-only

External link is report-only and was not fetched%3A https%3A//jemalloc.net/jemalloc.3.html#stats.retained. Owner%3A @apache/doris-website-maintainers

### Jemalloc Metadata memory is too large

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@
"description": "Doris 默认使用 Jemalloc 作为通用内存分配器,Jemalloc 自身占用的内存包括 Cache 和 Metadata 两部分,其中 Cache 包括 Thread Cache 和 Dirty Page 两部分,"
}
---

Check warning on line 8 in i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/trouble-shooting/memory-management/memory-analysis/jemalloc-memory-analysis.md

View workflow job for this annotation

GitHub Actions / Build Check

markdown-code-fence-language

Code fence should declare a language. Owner%3A @apache/doris-website-maintainers
Doris 默认使用 Jemalloc 作为通用内存分配器,Jemalloc 自身占用的内存包括 Cache 和 Metadata 两部分,其中 Cache 包括 Thread Cache 和 Dirty Page 两部分,在 http://{be_host}:{be_web_server_port}/memz 可以实时查看到内存分配器原始的 profile。
Doris 默认使用 Jemalloc 作为通用内存分配器,Jemalloc 自身占用的内存包括 Cache 和 Metadata 两部分,其中 Cache 包括 Thread Cache 和 Dirty Page 两部分,在 Doris BE 的 Process Profile Web 页面 http://{be_host}:{be_web_server_port}/profile 中 **Memory Info** 的 **Jemalloc Profiles** 部分可以实时查看到内存分配器原始的 profile。

## Jemalloc Cache 内存分析

Expand All @@ -20,13 +20,13 @@

BE 进程运行过程中,Jemalloc Cache 包括两部分。

- Thread Cache,在 Thread Cache 中缓存指定数量的 Page,参考 [Jemalloc opt.tcache](https://jemalloc.net/jemalloc.3.html#opt.tcache)。

Check notice on line 23 in i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/trouble-shooting/memory-management/memory-analysis/jemalloc-memory-analysis.md

View workflow job for this annotation

GitHub Actions / Build Check

link-external-report-only

External link is report-only and was not fetched%3A https%3A//jemalloc.net/jemalloc.3.html#opt.tcache. Owner%3A @apache/doris-website-maintainers

- Dirty Page,所有 Arena 中可以被复用的内存 Page。

### Jemalloc Cache 查看方法

查看 Doris BE 的 Web 页面 `http://{be_host}:{be_web_server_port}/memz`(webserver_port 默认 8040)可以获得 Jemalloc Profile,根据几组关键信息解读 Jemalloc Cache 的使用。
查看 Doris BE 的 Process Profile Web 页面 `http://{be_host}:{be_web_server_port}/profile`(webserver_port 默认 8040)中 **Memory Info** 的 **Jemalloc Profiles** 部分可以获得 Jemalloc Profile,根据几组关键信息解读 Jemalloc Cache 的使用。

- Jemalloc Profile 中的 `tcache_bytes`是 Jemalloc Thread Cache 的总字节数。如果 `tcache_bytes` 值较大,说明 Jemalloc Thread Cache 使用的内存过大。

Expand All @@ -39,7 +39,7 @@
考虑减小 `be.conf` 中 `JEMALLOC_CONF` 的 `lg_tcache_max`,`lg_tcache_max` 是允许缓存的 Page 字节大小上限,默认是 15,即 32 KB (2^15),超过这个大小的 Page 将不会缓存到 Thread Cache 中。`lg_tcache_max` 对应 Jemalloc Profile 中的 `Maximum thread-cached size class`。

> Doris 2.1 之前 `be.conf` 中 `JEMALLOC_CONF` 的 `lg_tcache_max` 默认是 20,在某些场景会导致 Jemalloc Cache 过大,Doris 2.1 之后已经改回了 Jemalloc 的默认值 15。

Check warning on line 42 in i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/trouble-shooting/memory-management/memory-analysis/jemalloc-memory-analysis.md

View workflow job for this annotation

GitHub Actions / Build Check

markdown-code-fence-language

Code fence should declare a language. Owner%3A @apache/doris-website-maintainers
这通常是 BE 进程中的查询或导入正在申请大量大 Size Class 的内存 Page,或者执行完一个大内存查询或导入后,Thread Cache 中缓存了大量大 Size Class 的内存 Page。Thread Cache 有两个清理时机,一是内存申请和释放到达一定次数时,回收长时间未使用的内存块;二是线程退出时回收全部 Page。此时存在一个 Bad Case,若线程后续一直没有执行新的查询或导入,从此不再分配内存,陷入一种所谓的 `idle` 状态。用户预期是查询结束后,内存是可以释放掉的,但实际上此场景下若线程没有退出,Thread Cache 并不会清理。

不过通常无需关注 Thread Cache,在进程可用内存不足时,若 Thread Cache 的大小超过 1G,Doris 将手动 Flush Thread Cache。
Expand All @@ -65,11 +65,11 @@
114688 14 1 102400 1 106496 15 1642496 17 1851392
```

减小 `be.conf` 中 `JEMALLOC_CONF` 的 `dirty_decay_ms` 到 2000 ms 或更小,`be.conf` 中默认 `dirty_decay_ms` 为 5000 ms。Jemalloc 会在 `dirty_decay_ms` 指定的时间内依照平滑梯度曲线释放 Dirty Page,参考 [Jemalloc opt.dirty_decay_ms](https://jemalloc.net/jemalloc.3.html#opt.dirty_decay_ms),当 BE 进程可用内存不足触发 Minor GC 或 Full GC 时会按照一定策略主动释放所有 Dirty Page。

Check notice on line 68 in i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/trouble-shooting/memory-management/memory-analysis/jemalloc-memory-analysis.md

View workflow job for this annotation

GitHub Actions / Build Check

link-external-report-only

External link is report-only and was not fetched%3A https%3A//jemalloc.net/jemalloc.3.html#opt.dirty_decay_ms. Owner%3A @apache/doris-website-maintainers

> Doris 2.1 之前 `be.conf` 中 `JEMALLOC_CONF` 的 `dirty_decay_ms` 默认是 15000,在某些场景会导致 Jemalloc Cache 过大,Doris 2.1 之后默认值是 5000。

Check warning on line 71 in i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/trouble-shooting/memory-management/memory-analysis/jemalloc-memory-analysis.md

View workflow job for this annotation

GitHub Actions / Build Check

markdown-code-fence-language

Code fence should declare a language. Owner%3A @apache/doris-website-maintainers
Jemalloc Profile 中的 `extents` 包含 Jemalloc 所有 `arena` 中不同 Page Size 的 Bucket 的统计值,其中 `ndirty` 是 Dirty Page 的个数,`dirty` 是 Dirty Page 的内存总和。参考 [Jemalloc](https://jemalloc.net/jemalloc.3.html) 中的 `stats.arenas.<i>.extents.<j>.{extent_type}_bytes` 将所有 Page Size 的 `dirty` 相加得到 Jemalloc 中 Dirty Page 的内存字节大小。

Check notice on line 72 in i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/trouble-shooting/memory-management/memory-analysis/jemalloc-memory-analysis.md

View workflow job for this annotation

GitHub Actions / Build Check

link-external-report-only

External link is report-only and was not fetched%3A https%3A//jemalloc.net/jemalloc.3.html. Owner%3A @apache/doris-website-maintainers

## Jemalloc Metadata 内存分析

Expand All @@ -83,7 +83,7 @@

### Jemalloc Metadata 查看方法

查看 Doris BE 的 Web 页面 `http://{be_host}:{be_web_server_port}/memz`(webserver_port 默认 8040)可以获得 Jemalloc Profile,查找 Jemalloc Profile 中关于 Jemalloc 整体的内存统计如下,其中 `metadata` 就是 Jemalloc Metadata 的内存大小。
查看 Doris BE 的 Process Profile Web 页面 `http://{be_host}:{be_web_server_port}/profile`(webserver_port 默认 8040)中 **Memory Info** 的 **Jemalloc Profiles** 部分可以获得 Jemalloc Profile,查找 Jemalloc Profile 中关于 Jemalloc 整体的内存统计如下,其中 `metadata` 就是 Jemalloc Metadata 的内存大小。

`Allocated: 2401232080, active: 2526302208, metadata: 535979296 (n_thp 221), resident: 2995621888, mapped: 3221979136, retained: 131542581248`

Expand All @@ -91,9 +91,9 @@

- `active` Jemalloc 为 BE 进程分配的所有 Page 总字节数,是 Page Size 的倍数,通常大于等于 `Allocated`。

- `metadata` Jemalloc 的元数据总字节数,和分配和缓存的 Page 个数、内存碎片 等因素都有关,参考文档 [Jemalloc stats.metadata](https://jemalloc.net/jemalloc.3.html#stats.metadata)

Check notice on line 94 in i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/trouble-shooting/memory-management/memory-analysis/jemalloc-memory-analysis.md

View workflow job for this annotation

GitHub Actions / Build Check

link-external-report-only

External link is report-only and was not fetched%3A https%3A//jemalloc.net/jemalloc.3.html#stats.metadata. Owner%3A @apache/doris-website-maintainers

- `retained` Jemalloc 保留的虚拟内存映射大小,也没有通过 munmap 或类似方法返回给操作系统,也没有强关联物理内存。参考文档 [Jemalloc stats.retained](https://jemalloc.net/jemalloc.3.html#stats.retained)

Check notice on line 96 in i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/trouble-shooting/memory-management/memory-analysis/jemalloc-memory-analysis.md

View workflow job for this annotation

GitHub Actions / Build Check

link-external-report-only

External link is report-only and was not fetched%3A https%3A//jemalloc.net/jemalloc.3.html#stats.retained. Owner%3A @apache/doris-website-maintainers

### Jemalloc Metadata 内存过大

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@
"description": "Doris 默认使用 Jemalloc 作为通用内存分配器,Jemalloc 自身占用的内存包括 Cache 和 Metadata 两部分,其中 Cache 包括 Thread Cache 和 Dirty Page 两部分,"
}
---

Check warning on line 8 in i18n/zh-CN/docusaurus-plugin-content-docs/version-3.x/admin-manual/trouble-shooting/memory-management/memory-analysis/jemalloc-memory-analysis.md

View workflow job for this annotation

GitHub Actions / Build Check

markdown-code-fence-language

Code fence should declare a language. Owner%3A @apache/doris-website-maintainers
Doris 默认使用 Jemalloc 作为通用内存分配器,Jemalloc 自身占用的内存包括 Cache 和 Metadata 两部分,其中 Cache 包括 Thread Cache 和 Dirty Page 两部分,在 http://{be_host}:{be_web_server_port}/memz 可以实时查看到内存分配器原始的 profile。
Doris 默认使用 Jemalloc 作为通用内存分配器,Jemalloc 自身占用的内存包括 Cache 和 Metadata 两部分,其中 Cache 包括 Thread Cache 和 Dirty Page 两部分,在 Doris BE 的 Process Profile Web 页面 http://{be_host}:{be_web_server_port}/profile 中 **Memory Info** 的 **Jemalloc Profiles** 部分可以实时查看到内存分配器原始的 profile。

## Jemalloc Cache 内存分析

Expand All @@ -26,7 +26,7 @@

### Jemalloc Cache 查看方法

查看 Doris BE 的 Web 页面 `http://{be_host}:{be_web_server_port}/memz`(webserver_port 默认 8040)可以获得 Jemalloc Profile,根据几组关键信息解读 Jemalloc Cache 的使用。
查看 Doris BE 的 Process Profile Web 页面 `http://{be_host}:{be_web_server_port}/profile`(webserver_port 默认 8040)中 **Memory Info** 的 **Jemalloc Profiles** 部分可以获得 Jemalloc Profile,根据几组关键信息解读 Jemalloc Cache 的使用。

- Jemalloc Profile 中的 `tcache_bytes`是 Jemalloc Thread Cache 的总字节数。如果 `tcache_bytes` 值较大,说明 Jemalloc Thread Cache 使用的内存过大。

Expand All @@ -39,7 +39,7 @@
考虑减小 `be.conf` 中 `JEMALLOC_CONF` 的 `lg_tcache_max`,`lg_tcache_max` 是允许缓存的 Page 字节大小上限,默认是 15,即 32 KB (2^15),超过这个大小的 Page 将不会缓存到 Thread Cache 中。`lg_tcache_max` 对应 Jemalloc Profile 中的 `Maximum thread-cached size class`。

> Doris 2.1 之前 `be.conf` 中 `JEMALLOC_CONF` 的 `lg_tcache_max` 默认是 20,在某些场景会导致 Jemalloc Cache 过大,Doris 2.1 之后已经改回了 Jemalloc 的默认值 15。

Check warning on line 42 in i18n/zh-CN/docusaurus-plugin-content-docs/version-3.x/admin-manual/trouble-shooting/memory-management/memory-analysis/jemalloc-memory-analysis.md

View workflow job for this annotation

GitHub Actions / Build Check

markdown-code-fence-language

Code fence should declare a language. Owner%3A @apache/doris-website-maintainers
这通常是 BE 进程中的查询或导入正在申请大量大 Size Class 的内存 Page,或者执行完一个大内存查询或导入后,Thread Cache 中缓存了大量大 Size Class 的内存 Page。Thread Cache 有两个清理时机,一是内存申请和释放到达一定次数时,回收长时间未使用的内存块;二是线程退出时回收全部 Page。此时存在一个 Bad Case,若线程后续一直没有执行新的查询或导入,从此不再分配内存,陷入一种所谓的 `idle` 状态。用户预期是查询结束后,内存是可以释放掉的,但实际上此场景下若线程没有退出,Thread Cache 并不会清理。

不过通常无需关注 Thread Cache,在进程可用内存不足时,若 Thread Cache 的大小超过 1G,Doris 将手动 Flush Thread Cache。
Expand Down Expand Up @@ -68,7 +68,7 @@
减小 `be.conf` 中 `JEMALLOC_CONF` 的 `dirty_decay_ms` 到 2000 ms 或更小,`be.conf` 中默认 `dirty_decay_ms` 为 5000 ms。Jemalloc 会在 `dirty_decay_ms` 指定的时间内依照平滑梯度曲线释放 Dirty Page,参考 [Jemalloc opt.dirty_decay_ms](https://jemalloc.net/jemalloc.3.html#opt.dirty_decay_ms),当 BE 进程可用内存不足触发 Minor GC 或 Full GC 时会按照一定策略主动释放所有 Dirty Page。

> Doris 2.1 之前 `be.conf` 中 `JEMALLOC_CONF` 的 `dirty_decay_ms` 默认是 15000,在某些场景会导致 Jemalloc Cache 过大,Doris 2.1 之后默认值是 5000。

Check warning on line 71 in i18n/zh-CN/docusaurus-plugin-content-docs/version-3.x/admin-manual/trouble-shooting/memory-management/memory-analysis/jemalloc-memory-analysis.md

View workflow job for this annotation

GitHub Actions / Build Check

markdown-code-fence-language

Code fence should declare a language. Owner%3A @apache/doris-website-maintainers
Jemalloc Profile 中的 `extents` 包含 Jemalloc 所有 `arena` 中不同 Page Size 的 Bucket 的统计值,其中 `ndirty` 是 Dirty Page 的个数,`dirty` 是 Dirty Page 的内存总和。参考 [Jemalloc](https://jemalloc.net/jemalloc.3.html) 中的 `stats.arenas.<i>.extents.<j>.{extent_type}_bytes` 将所有 Page Size 的 `dirty` 相加得到 Jemalloc 中 Dirty Page 的内存字节大小。

## Jemalloc Metadata 内存分析
Expand All @@ -83,7 +83,7 @@

### Jemalloc Metadata 查看方法

查看 Doris BE 的 Web 页面 `http://{be_host}:{be_web_server_port}/memz`(webserver_port 默认 8040)可以获得 Jemalloc Profile,查找 Jemalloc Profile 中关于 Jemalloc 整体的内存统计如下,其中 `metadata` 就是 Jemalloc Metadata 的内存大小。
查看 Doris BE 的 Process Profile Web 页面 `http://{be_host}:{be_web_server_port}/profile`(webserver_port 默认 8040)中 **Memory Info** 的 **Jemalloc Profiles** 部分可以获得 Jemalloc Profile,查找 Jemalloc Profile 中关于 Jemalloc 整体的内存统计如下,其中 `metadata` 就是 Jemalloc Metadata 的内存大小。

`Allocated: 2401232080, active: 2526302208, metadata: 535979296 (n_thp 221), resident: 2995621888, mapped: 3221979136, retained: 131542581248`

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@
"description": "Doris 默认使用 Jemalloc 作为通用内存分配器,Jemalloc 自身占用的内存包括 Cache 和 Metadata 两部分,其中 Cache 包括 Thread Cache 和 Dirty Page 两部分,"
}
---

Check warning on line 8 in i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/admin-manual/trouble-shooting/memory-management/memory-analysis/jemalloc-memory-analysis.md

View workflow job for this annotation

GitHub Actions / Build Check

markdown-code-fence-language

Code fence should declare a language. Owner%3A @apache/doris-website-maintainers
Doris 默认使用 Jemalloc 作为通用内存分配器,Jemalloc 自身占用的内存包括 Cache 和 Metadata 两部分,其中 Cache 包括 Thread Cache 和 Dirty Page 两部分,在 http://{be_host}:{be_web_server_port}/memz 可以实时查看到内存分配器原始的 profile。
Doris 默认使用 Jemalloc 作为通用内存分配器,Jemalloc 自身占用的内存包括 Cache 和 Metadata 两部分,其中 Cache 包括 Thread Cache 和 Dirty Page 两部分,在 Doris BE 的 Process Profile Web 页面 http://{be_host}:{be_web_server_port}/profile 中 **Memory Info** 的 **Jemalloc Profiles** 部分可以实时查看到内存分配器原始的 profile。

## Jemalloc Cache 内存分析

Expand All @@ -26,7 +26,7 @@

### Jemalloc Cache 查看方法

查看 Doris BE 的 Web 页面 `http://{be_host}:{be_web_server_port}/memz`(webserver_port 默认 8040)可以获得 Jemalloc Profile,根据几组关键信息解读 Jemalloc Cache 的使用。
查看 Doris BE 的 Process Profile Web 页面 `http://{be_host}:{be_web_server_port}/profile`(webserver_port 默认 8040)中 **Memory Info** 的 **Jemalloc Profiles** 部分可以获得 Jemalloc Profile,根据几组关键信息解读 Jemalloc Cache 的使用。

- Jemalloc Profile 中的 `tcache_bytes`是 Jemalloc Thread Cache 的总字节数。如果 `tcache_bytes` 值较大,说明 Jemalloc Thread Cache 使用的内存过大。

Expand Down Expand Up @@ -83,7 +83,7 @@

### Jemalloc Metadata 查看方法

查看 Doris BE 的 Web 页面 `http://{be_host}:{be_web_server_port}/memz`(webserver_port 默认 8040)可以获得 Jemalloc Profile,查找 Jemalloc Profile 中关于 Jemalloc 整体的内存统计如下,其中 `metadata` 就是 Jemalloc Metadata 的内存大小。
查看 Doris BE 的 Process Profile Web 页面 `http://{be_host}:{be_web_server_port}/profile`(webserver_port 默认 8040)中 **Memory Info** 的 **Jemalloc Profiles** 部分可以获得 Jemalloc Profile,查找 Jemalloc Profile 中关于 Jemalloc 整体的内存统计如下,其中 `metadata` 就是 Jemalloc Metadata 的内存大小。

`Allocated: 2401232080, active: 2526302208, metadata: 535979296 (n_thp 221), resident: 2995621888, mapped: 3221979136, retained: 131542581248`

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
}
---

Doris uses Jemalloc as the general memory allocator by default. The memory occupied by Jemalloc itself includes Cache and Metadata. Cache includes Thread Cache and Dirty Page. You can view the original profile of the memory allocator in real time at http://{be_host}:{be_web_server_port}/memz.
Doris uses Jemalloc as the general memory allocator by default. The memory occupied by Jemalloc itself includes Cache and Metadata. Cache includes Thread Cache and Dirty Page. You can view the original profile of the memory allocator in real time in the **Jemalloc Profiles** section under **Memory Info** on Doris BE's Process Profile Web page http://{be_host}:{be_web_server_port}/profile.

## Jemalloc Cache Memory Analysis

Expand All @@ -26,7 +26,7 @@ During the running of the BE process, Jemalloc Cache consists of two parts.

### Jemalloc Cache View Method

View Doris BE's Web page `http://{be_host}:{be_web_server_port}/memz` (webserver_port defaults to 8040) to obtain Jemalloc Profile, and interpret the use of Jemalloc Cache based on several sets of key information.
View the **Jemalloc Profiles** section under **Memory Info** on Doris BE's Process Profile Web page `http://{be_host}:{be_web_server_port}/profile` (webserver_port defaults to 8040) to obtain the Jemalloc Profile, and interpret the use of Jemalloc Cache based on several sets of key information.

- `tcache_bytes` in Jemalloc Profile is the total number of bytes of Jemalloc Thread Cache. If the `tcache_bytes` value is large, it means that the memory used by Jemalloc Thread Cache is too large.

Expand Down Expand Up @@ -82,7 +82,7 @@ MemTrackerLimiter Label=tc/jemalloc_metadata, Type=overview, Limit=-1.00 B(-1 B)

### How to view Jemalloc Metadata

You can get the Jemalloc Profile by viewing the Doris BE web page `http://{be_host}:{be_web_server_port}/memz` (webserver_port defaults to 8040). Find the overall memory statistics of Jemalloc in the Jemalloc Profile as follows, where `metadata` is the memory size of Jemalloc Metadata.
You can get the Jemalloc Profile from the **Jemalloc Profiles** section under **Memory Info** on the Doris BE Process Profile web page `http://{be_host}:{be_web_server_port}/profile` (webserver_port defaults to 8040). Find the overall memory statistics of Jemalloc in the Jemalloc Profile as follows, where `metadata` is the memory size of Jemalloc Metadata.

`Allocated: 2401232080, active: 2526302208, metadata: 535979296 (n_thp 221), resident: 2995621888, mapped: 3221979136, retained: 131542581248`

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
}
---

Doris uses Jemalloc as the general memory allocator by default. The memory occupied by Jemalloc itself includes Cache and Metadata. Cache includes Thread Cache and Dirty Page. You can view the original profile of the memory allocator in real time at http://{be_host}:{be_web_server_port}/memz.
Doris uses Jemalloc as the general memory allocator by default. The memory occupied by Jemalloc itself includes Cache and Metadata. Cache includes Thread Cache and Dirty Page. You can view the original profile of the memory allocator in real time in the **Jemalloc Profiles** section under **Memory Info** on Doris BE's Process Profile Web page http://{be_host}:{be_web_server_port}/profile.

## Jemalloc Cache Memory Analysis

Expand All @@ -26,7 +26,7 @@ During the running of the BE process, Jemalloc Cache consists of two parts.

### Jemalloc Cache View Method

View Doris BE's Web page `http://{be_host}:{be_web_server_port}/memz` (webserver_port defaults to 8040) to obtain Jemalloc Profile, and interpret the use of Jemalloc Cache based on several sets of key information.
View the **Jemalloc Profiles** section under **Memory Info** on Doris BE's Process Profile Web page `http://{be_host}:{be_web_server_port}/profile` (webserver_port defaults to 8040) to obtain the Jemalloc Profile, and interpret the use of Jemalloc Cache based on several sets of key information.

- `tcache_bytes` in Jemalloc Profile is the total number of bytes of Jemalloc Thread Cache. If the `tcache_bytes` value is large, it means that the memory used by Jemalloc Thread Cache is too large.

Expand Down Expand Up @@ -82,7 +82,7 @@ MemTrackerLimiter Label=tc/jemalloc_metadata, Type=overview, Limit=-1.00 B(-1 B)

### How to view Jemalloc Metadata

You can get the Jemalloc Profile by viewing the Doris BE web page `http://{be_host}:{be_web_server_port}/memz` (webserver_port defaults to 8040). Find the overall memory statistics of Jemalloc in the Jemalloc Profile as follows, where `metadata` is the memory size of Jemalloc Metadata.
You can get the Jemalloc Profile from the **Jemalloc Profiles** section under **Memory Info** on the Doris BE Process Profile web page `http://{be_host}:{be_web_server_port}/profile` (webserver_port defaults to 8040). Find the overall memory statistics of Jemalloc in the Jemalloc Profile as follows, where `metadata` is the memory size of Jemalloc Metadata.

`Allocated: 2401232080, active: 2526302208, metadata: 535979296 (n_thp 221), resident: 2995621888, mapped: 3221979136, retained: 131542581248`

Expand Down
Loading