Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion .bazelignore
Original file line number Diff line number Diff line change
@@ -1 +1,8 @@
./example/build_with_bazel
./example/build_with_bazel

# `registry/` is brpc's self-maintained Bzlmod registry. Its overlay
# BUILD.bazel files reference sources from the libunwind tarball that is
# only materialized at module resolution time, not in the source tree.
# Without this entry, `bazel build //...` would try to evaluate them as
# regular packages and fail with "missing input file ...".
./registry
3 changes: 3 additions & 0 deletions .licenserc.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -218,4 +218,7 @@ header:
# Fuzzing seed
- 'test/fuzzing/fuzz_*_seed_corpus/*'

# bazel-central-registry
- 'registry/**'

comment: on-failure
38 changes: 38 additions & 0 deletions LICENSE
Original file line number Diff line number Diff line change
Expand Up @@ -959,3 +959,41 @@ copyright The Chromium Authors and licensed under the 3-clause BSD license:
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

--------------------------------------------------------------------------------

registry/modules/libunwind/**: licensed under the following terms:

Forked from the Bazel Central Registry (BCR):
https://github.com/bazelbuild/bazel-central-registry/tree/main/modules/libunwind

The upstream files are distributed under the Apache License, Version 2.0,
the same license as brpc itself (the full text of which is reproduced at
the top of this LICENSE file). brpc has modified the forked files; per the
Apache License 2.0 §4(b), the modifications are summarized below:

- registry/modules/libunwind/<version>/MODULE.bazel
- registry/modules/libunwind/<version>/overlay/MODULE.bazel
Append the suffix `.brpc-no-unwind` to the `version` field, marking
these as brpc's variant of the libunwind module.

- registry/modules/libunwind/<version>/overlay/BUILD.bazel
Add the `hide_unwind_symbols` config_setting (gated by
`--define=libunwind_hide_unwind_symbols=true`); when the switch is
on, drop `src/unwind/*.c` (the GCC `_Unwind_*` ABI compatibility
layer) from `unwind_srcs` so the resulting libunwind does not export
`_Unwind_*` symbols. See docs/cn/bthread_tracer.md for the
rationale.

- registry/modules/libunwind/<version>/source.json
Update overlay file SHA-256 hashes to match the modified
BUILD.bazel / MODULE.bazel.

- registry/modules/libunwind/metadata.json
Replace the `versions` array with brpc's renamed versions.

The above only governs the build/source files brpc redistributes inside
registry/modules/libunwind/. The libunwind source code itself, downloaded
by Bazel from the URL pinned in source.json, remains governed by the
libunwind project's own license; see https://github.com/libunwind/libunwind
for details.
1 change: 1 addition & 0 deletions bazel/config/BUILD.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -136,6 +136,7 @@ config_setting(
define_values = {
"with_bthread_tracer": "true",
},
visibility = ["//visibility:public"],
)

config_setting(
Expand Down
141 changes: 140 additions & 1 deletion docs/cn/bthread_tracer.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ jump_stack是bthread挂起或者运行的必经之路,也是STB的拦截点。

# 使用方法

1. 下载安装libunwind和abseil-cpp。
1. 下载安装libunwind和abseil-cpp。**注意:libunwind 必须从源码编译,不要使用系统包管理器安装的 `libunwind-dev` / `libunwind-devel`**,否则会触发本文末尾「[已知问题:libunwind 与 libgcc_s 的 `_Unwind_*` 符号冲突](#已知问题libunwind-与-libgcc_s-的-_unwind_-符号冲突)」中描述的崩溃。bazel 构建可以跳过此步,直接使用 brpc 仓库自维护的 libunwind 版本。
2. 给config_brpc.sh增加`--with-bthread-tracer`选项或者给cmake增加`-DWITH_BTHREAD_TRACER=ON`选项或者给bazel(Bzlmod模式)增加`--define with_bthread_tracer=true`选项。
3. 访问服务的内置服务:`http://ip:port/bthreads/<bthread_id>?st=1`或者代码里调用`bthread::stack_trace()`函数。
4. 如果希望追踪pthread的调用栈,在对应pthread上调用`bthread::init_for_pthread_stack_trace()`函数获取一个伪bthread_t,然后使用步骤3即可获取pthread调用栈。
Expand All @@ -73,6 +73,145 @@ jump_stack是bthread挂起或者运行的必经之路,也是STB的拦截点。
#5 0x00007fdbbfa58dc0 bthread::TaskGroup::task_runner()
```

# 已知问题

## libunwind 与 libgcc_s 的 `_Unwind_*` 符号冲突

### 现象

启用 bthread tracer 后,可能在 `bthread_exit` / `pthread_exit` 或者 C++ 异常处理路径上偶发段错误,类似如下调用栈:

```text
#0 0x0000000000000000 in ?? ()
#1 0x00007fa2b5d6458a in _ULx86_64_dwarf_find_proc_info ()
from /root/.cache/bazel/_bazel_root/743b333b2429a1dbd390ef66b59c771d/execroot/_main/bazel-out/k8-fastbuild/bin/test/../_solib_k8/libexternal_Slibunwind~_Slibunwind.so
#2 0x00007fa2b5d6668d in fetch_proc_info ()
from /root/.cache/bazel/_bazel_root/743b333b2429a1dbd390ef66b59c771d/execroot/_main/bazel-out/k8-fastbuild/bin/test/../_solib_k8/libexternal_Slibunwind~_Slibunwind.so
#3 0x00007fa2b5d681a1 in _ULx86_64_dwarf_make_proc_info ()
from /root/.cache/bazel/_bazel_root/743b333b2429a1dbd390ef66b59c771d/execroot/_main/bazel-out/k8-fastbuild/bin/test/../_solib_k8/libexternal_Slibunwind~_Slibunwind.so
#4 0x00007fa2b5d70cfd in _ULx86_64_get_proc_info ()
from /root/.cache/bazel/_bazel_root/743b333b2429a1dbd390ef66b59c771d/execroot/_main/bazel-out/k8-fastbuild/bin/test/../_solib_k8/libexternal_Slibunwind~_Slibunwind.so
#5 0x00007fa2b5d6c775 in __libunwind_Unwind_GetLanguageSpecificData ()
from /root/.cache/bazel/_bazel_root/743b333b2429a1dbd390ef66b59c771d/execroot/_main/bazel-out/k8-fastbuild/bin/test/../_solib_k8/libexternal_Slibunwind~_Slibunwind.so
#6 0x00007fa2b503c6df in __gxx_personality_v0 () from /lib/x86_64-linux-gnu/libstdc++.so.6
#7 0x00007fa2b5452ce5 in ?? () from /lib/x86_64-linux-gnu/libgcc_s.so.1
#8 0x00007fa2b54533c0 in _Unwind_ForcedUnwind () from /lib/x86_64-linux-gnu/libgcc_s.so.1
#9 0x00007fa2b4ca57a4 in __GI___pthread_unwind (buf=<optimized out>) at ./nptl/unwind.c:130
#10 0x00007fa2b4c9dd22 in __do_cancel () at ../sysdeps/nptl/pthreadP.h:271
#11 __GI___pthread_exit (value=0x0) at ./nptl/pthread_exit.c:36
#12 0x0000000000000000 in ?? ()
```

### 根因

libunwind 的 `src/unwind/*.c` 实现了 GCC 的 `_Unwind_*` ABI 兼容层(`_Unwind_GetLanguageSpecificData`、`_Unwind_ForcedUnwind`、`_Unwind_Resume` 等),与 `libgcc_s.so.1` 提供同名的全局符号。当 libunwind 以**动态库**形式被链接,并且在最终二进制的 `DT_NEEDED` 列表中位置比 `libgcc_s.so.1` 靠前时,运行时动态链接器 `ld.so` 会把 `pthread_exit` / 异常处理触发的 `_Unwind_*` 调用解析到 libunwind 中的 DWARF 实现。该实现需要的内部上下文在 `pthread_exit` 路径上未被 bRPC 初始化好,从而触发空指针访问。

这是一个 ELF **运行时符号解析顺序**问题,与编译器(GCC / Clang)无关 —— Clang 默认运行时同样使用 `libstdc++ + libgcc_s`,会复现完全一致的崩溃。

### 解决方案

> **重要:不要使用系统包管理器安装的 libunwind**(例如 `apt install libunwind-dev`、`yum install libunwind-devel`)。多数发行版打包的 `libunwind.so` 仍把 `_Unwind_*` 暴露在动态符号表中,会触发本节描述的崩溃。
>
> 必须使用**从源码编译的 libunwind**。上游 `./configure` + `make` 默认会通过 `-Wl,--version-script` 把 `_Unwind_*` 标为 local,不导出到动态符号表,从而避免冲突。

下表汇总了三种构建方式的推荐方案:

| 构建方式 | 推荐方案 |
|---|---|
| `config_brpc.sh` + `make` | 从源码编译并安装 libunwind,把头文件与库目录显式传给 `config_brpc.sh` |
| `cmake` | 从源码编译并安装 libunwind,把头文件与库目录显式传给 `cmake` |
| `bazel`(Bzlmod) | 直接使用 brpc 仓库自维护的 libunwind 版本 |

### `make`(`config_brpc.sh`)

源码编译并安装 libunwind 到一个独立目录(避免污染系统目录),然后让 `config_brpc.sh` 显式从该目录查找 libunwind。

```bash
# 1) 源码编译 libunwind(推荐 v1.8.1 或以上版本)
git clone https://github.com/libunwind/libunwind.git
cd libunwind && git checkout tags/v1.8.1
mkdir -p /opt/libunwind
autoreconf -i
./configure --prefix=/opt/libunwind
make -j$(nproc) && make install
cd ..

# 2) 让 config_brpc.sh 使用 /libunwind 下的头文件与库(不要让它自动找到系统的 libunwind-dev)
cd brpc
sh config_brpc.sh \
--with-bthread-tracer \
--headers="/opt/libunwind/include /usr/include" \
--libs="/opt/libunwind/lib /usr/lib /usr/lib64"
make -j$(nproc)
```

构建完成后可用以下命令确认 libunwind.so 没有导出 `_Unwind_*`:

```bash
nm -D /libunwind/lib/libunwind.so | grep ' T _Unwind_' \
&& echo "WARN: _Unwind_* exported" \
|| echo "OK: _Unwind_* hidden"
```

### `cmake`

[`CMakeLists.txt`](../../CMakeLists.txt:90-100) 通过 `find_library(... NAMES unwind unwind-x86_64)` 查找 libunwind。同样需要先源码编译 libunwind 到独立前缀,再用 `CMAKE_PREFIX_PATH` 让 cmake 优先在该前缀下查找:

```bash
# 1) 源码编译 libunwind(同 make 章节)

# 2) 让 cmake 在 /libunwind 下优先查找头文件和库
cd brpc
mkdir build && cd build
cmake -DWITH_BTHREAD_TRACER=ON \
-DCMAKE_PREFIX_PATH=/opt/libunwind \
..
make -j$(nproc)
```

> 提示:如果系统已经装了 `libunwind-dev`,`find_library` 仍可能优先匹配到 `/usr/lib`。可在 cmake 命令上额外指定
> `-DLIBUNWIND_LIB=/libunwind/lib/libunwind.so -DLIBUNWIND_X86_64_LIB=/libunwind/lib/libunwind-x86_64.so -DLIBUNWIND_INCLUDE_PATH=/libunwind/include`
> 强制走自编译版本,避免系统包混入。

### `bazel`(Bzlmod)

bRPC 仓库已经在 [`registry/modules/libunwind/`](../../registry/modules/libunwind/) 维护了一份 libunwind 的 Bzlmod overlay,并通过 [`.bazelrc`](../../.bazelrc) 中的 `--registry=https://github.com/apache/brpc/registry` 使用 bRPC 维护的 overlay。版本号采用 `<base>.brpc-no-unwind` 后缀(例如 `1.8.3.brpc-no-unwind`),用以区别于 BCR 上的同基版本。该 overlay 增加了一个开关:

```
--define libunwind_hide_unwind_symbols=true
```

开启后,libunwind 的 `src/unwind/*.c`(即 GCC `_Unwind_*` 兼容层)整体不参与编译,等效于上游 autoconf 默认效果。bRPC 只使用 libunwind 的 `unw_*` 原生 API(`unw_getcontext`、`unw_init_local`、`unw_step` 等),不依赖 `_Unwind_*` 兼容层,因此该开关安全无副作用。

`.bazelrc` 已默认在 `build:test` / `test` 配置下打开此开关:

```
build:test --define libunwind_hide_unwind_symbols=true
test --define libunwind_hide_unwind_symbols=true
```

用户按文档使用方法第 2 步加上 `--define=with_bthread_tracer=true` 即可:

```bash
# 测试场景,.bazelrc 中 test 配置已自动带上 hide 开关
bazel test //test:bthread_unittest

# 非测试构建(生产部署),需要显式同时带上两个 define
bazel build --define=with_bthread_tracer=true \
--define=libunwind_hide_unwind_symbols=true \
//...
```

> **特别注意**:如果在生产构建中只启用 `--define=with_bthread_tracer=true` 而漏掉 `--define=libunwind_hide_unwind_symbols=true`,binary 在 `pthread_exit` / 异常路径上会概率性崩溃。

构建后可用以下命令验证 libunwind 共享库没有导出 `_Unwind_*`:

```bash
nm -D bazel-bin/external/_solib_*/libexternal*libunwind*.so 2>/dev/null \
| grep ' T _Unwind_' || echo "OK: no _Unwind_* exported by libunwind.so"
```


# 相关flag

- `signal_trace_timeout_ms`:信号追踪模式的超时时间,默认为50ms。
21 changes: 21 additions & 0 deletions registry/modules/libunwind/1.8.1.brpc-no-unwind/MODULE.bazel
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# Forked from the Bazel Central Registry:
# https://github.com/bazelbuild/bazel-central-registry/blob/main/modules/libunwind/1.8.1/MODULE.bazel
# Distributed under the Apache License, Version 2.0. See the `registry/modules/
# libunwind/**` entry near the bottom of brpc's LICENSE file.
#
# brpc modification (Apache License 2.0 §4(b)):
# - `version` suffixed with `.brpc-no-unwind` to distinguish this brpc
# variant from the upstream BCR version.

module(
name = "libunwind",
version = "1.8.1.brpc-no-unwind",
bazel_compatibility = [">=7.2.1"], # need support for "overlay" directory
compatibility_level = 0,
)

bazel_dep(name = "bazel_skylib", version = "1.7.1")
bazel_dep(name = "platforms", version = "0.0.11")
bazel_dep(name = "rules_cc", version = "0.1.1")
bazel_dep(name = "xz", version = "5.4.5.bcr.5")
bazel_dep(name = "zlib", version = "1.3.1.bcr.5")
Loading
Loading