Addr2line enhance 2

samples/debug-tools/README.md

@@ -52,25 +52,104 @@ $ python3 ../../../test-tools/addr2line/addr2line.py \
     call_stack.txt
 ```
 
+This sample also ships `symbolicate.sh` (runs the three modes shown below in
+sequence) and `verify.sh` (asserts inline expansion appears in both the
+classic-interp and AOT outputs — used by CI, useful locally as a smoke test).
+Both expect to be run from the sample root after building:
+
+```bash
+$ ./symbolicate.sh
+$ ./verify.sh
+```
+
 The output should be something like:
 
 ```text
-0: c
-        at wasm-micro-runtime/samples/debug-tools/wasm-apps/trap.c:5:1
+0: trap_helper (inlined into c)
+	at /path/to/wasm-micro-runtime/samples/debug-tools/wasm-apps/trap.c:9:5
+   c
+	at /path/to/wasm-micro-runtime/samples/debug-tools/wasm-apps/trap.c:16:12
 1: b
-        at wasm-micro-runtime/samples/debug-tools/wasm-apps/trap.c:11:12
+	at /path/to/wasm-micro-runtime/samples/debug-tools/wasm-apps/trap.c:23:12
 2: a
-        at wasm-micro-runtime/samples/debug-tools/wasm-apps/trap.c:17:12
+	at /path/to/wasm-micro-runtime/samples/debug-tools/wasm-apps/trap.c:29:12
 3: main
-        at wasm-micro-runtime/samples/debug-tools/wasm-apps/trap.c:24:5
+	at /path/to/wasm-micro-runtime/samples/debug-tools/wasm-apps/trap.c:36:5
 4: __main_void
-        at unknown:?:?
+	at ??:0
 5: _start
+	at wasisdk://v25.0/build/sysroot/wasi-libc-wasm32-wasi/libc-bottom-half/crt/crt1-command.c:43:13
 ```
 
-If WAMR is run in fast interpreter mode (`WAMR_BUILD_FAST_INTERP=1`), addresses in the stack trace cannot be tracked back to location info.
-If WAMR <= `1.3.2` is used, the stack trace does not contain addresses.
-In those two cases, run the script with `--no-addr`: the line info returned refers to the start of the function
+Frame `0` shows **inline expansion** in action — `trap_helper` (the actual trap site)
+and `c` (its caller) appear together under index `0` because `trap_helper` was inlined
+into `c` by `__attribute__((always_inline))`. The `(inlined into c)` suffix on
+`trap_helper` makes the relationship explicit: the runtime saw one WASM frame for `c`,
+but addr2line.py reconstructs the full source-level call chain from DWARF
+`DW_TAG_inlined_subroutine` entries. Each frame in the chain except the outermost
+gets the `(inlined into <next>)` annotation.
+
+### Inline expansion
+
+addr2line.py automatically expands inline call chains when the binary's DWARF contains
+`DW_TAG_inlined_subroutine` entries. This happens when functions are inlined either by
+optimization (e.g. `-O2`, `-Oz`) or by `__attribute__((always_inline))`.
+
+The included `trap.c` marks `trap_helper` as always_inline and places `__builtin_trap()`
+inside it. The trap address therefore falls within the inlined region, producing the
+multi-line frame `0` shown above. Each inlined frame's source location is reported
+independently, so you can see exactly which inlined call chain led to the trap.
+
+Inline expansion requires no flags — it's automatic whenever the DWARF carries the
+relevant inline metadata.
+
+For details on the tools `addr2line.py` orchestrates internally and why each
+is needed, see [`test-tools/addr2line/README.md`](../../test-tools/addr2line/README.md).
+The longer-form discussion of the interval-table overlay — applied to correct
+the LLVM symbolizer's outermost function-name lookup on wasm — also lives there.
+
+### Execution modes — `--mode={interp,aot,fast-interp}`
+
+Different WAMR execution modes report call-stack offsets in different address spaces.
+Pass the right `--mode` to addr2line.py so offsets are interpreted correctly:
+
+| Mode | When | Adjustment | Source resolution |
+|------|------|-----------|-------------------|
+| `interp` (default) | Classic interpreter (`WAMR_BUILD_FAST_INTERP=0`) | `offset - code_start - 1` (post-advance) | Full file:line, inline expansion |
+| `aot` | AOT-compiled .aot files | `offset - code_start` (wamrc commits at instruction start) | Full file:line, inline expansion |
+| `fast-interp` | Fast interpreter (`WAMR_BUILD_FAST_INTERP=1`) | (not mappable) | Function-name only |
+
+For AOT call stacks:
+
+```bash
+$ python3 ../../../test-tools/addr2line/addr2line.py \
+    --wasi-sdk /opt/wasi-sdk \
+    --wabt /opt/wabt \
+    --wasm-file wasm-apps/trap.wasm \
+    --mode aot \
+    call_stack_aot.txt
+```
+
+For fast-interp call stacks:
+
+```bash
+$ python3 ../../../test-tools/addr2line/addr2line.py \
+    --wasi-sdk /opt/wasi-sdk \
+    --wabt /opt/wabt \
+    --wasm-file wasm-apps/trap.wasm \
+    --mode fast-interp \
+    call_stack.txt
+```
+
+Fast-interp transforms the WASM bytecode in-memory at load time (replaces opcodes
+with handler indices, expands LEBs to fixed widths). The runtime ip therefore
+points into the *transformed* buffer — there's no way to map an offset back to a
+source line. `--mode=fast-interp` falls back to function-name lookup (equivalent
+to `--no-addr`).
+
+If WAMR <= `1.3.2` is used, the stack trace does not contain addresses at all.
+In that case, run the script with `--no-addr`: the line info returned refers to
+the start of the function.
 
 ```bash
 $ python3 ../../../test-tools/addr2line/addr2line.py \

samples/debug-tools/symbolicate.sh

@@ -1,23 +1,52 @@
 #!/bin/bash
-set -euox pipefail
-
-# Symbolicate .wasm
-python3 ../../../test-tools/addr2line/addr2line.py \
-    --wasi-sdk /opt/wasi-sdk \
-    --wabt /opt/wabt \
-    --wasm-file wasm-apps/trap.wasm \
-    call_stack.txt
-
-# Symbolicate .wasm with `--no-addr`
-python3 ../../../test-tools/addr2line/addr2line.py \
-    --wasi-sdk /opt/wasi-sdk \
-    --wabt /opt/wabt \
-    --wasm-file wasm-apps/trap.wasm \
-    call_stack.txt --no-addr
-
-# Symbolicate .aot
-python3 ../../../test-tools/addr2line/addr2line.py \
-    --wasi-sdk /opt/wasi-sdk \
-    --wabt /opt/wabt \
-    --wasm-file wasm-apps/trap.wasm \
-    call_stack_aot.txt
+# Copyright (C) 2019 Intel Corporation.  All rights reserved.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+set -euo pipefail
+
+# Symbolicate the captured call stack files (call_stack.txt for .wasm,
+# call_stack_aot.txt for .aot) located in this sample's build/ directory.
+# Self-locating so invocation works from any cwd.
+
+SCRIPT_DIR=$(cd "$(dirname "$0")" && pwd)
+BUILD_DIR="${SCRIPT_DIR}/build"
+ADDR2LINE="${SCRIPT_DIR}/../../test-tools/addr2line/addr2line.py"
+WASM_FILE="${BUILD_DIR}/wasm-apps/trap.wasm"
+CALL_STACK="${BUILD_DIR}/call_stack.txt"
+CALL_STACK_AOT="${BUILD_DIR}/call_stack_aot.txt"
+
+WASI_SDK_PATH="${WASI_SDK_PATH:-/opt/wasi-sdk}"
+WABT_PATH="${WABT_PATH:-/opt/wabt}"
+
+banner() {
+    echo
+    echo "===== $* ====="
+}
+
+# .wasm + classic-interp: addr2line.py defaults to --mode=interp.
+banner "wasm / --mode=interp (default; classic interpreter, post-advance offsets)"
+python3 "$ADDR2LINE" \
+    --wasi-sdk "$WASI_SDK_PATH" \
+    --wabt "$WABT_PATH" \
+    --wasm-file "$WASM_FILE" \
+    "$CALL_STACK"
+
+# Same call stack with --no-addr: function-level resolution only — use
+# this for fast-interp call stacks (transformed bytecode has no source
+# mapping) and for very old iwasm dumps that don't carry offsets.
+banner "wasm / --no-addr (function-name lookup only; fast-interp & WAMR<=1.3.2 fallback)"
+python3 "$ADDR2LINE" \
+    --wasi-sdk "$WASI_SDK_PATH" \
+    --wabt "$WABT_PATH" \
+    --wasm-file "$WASM_FILE" \
+    --no-addr \
+    "$CALL_STACK"
+
+# .aot: --mode=aot — wamrc commits ip at instruction-start (not post-advance).
+banner "aot / --mode=aot (instruction-start offsets)"
+python3 "$ADDR2LINE" \
+    --wasi-sdk "$WASI_SDK_PATH" \
+    --wabt "$WABT_PATH" \
+    --wasm-file "$WASM_FILE" \
+    --mode aot \
+    "$CALL_STACK_AOT"

samples/debug-tools/verify.sh

@@ -0,0 +1,83 @@
+#!/bin/bash
+# Copyright (C) 2019 Intel Corporation.  All rights reserved.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+#
+# Verify symbolicated output for samples/debug-tools.
+#
+# Usage: ./verify.sh
+#
+# Captures call stacks for both .wasm and .aot, runs symbolicate.sh, and
+# asserts that the inline expansion of trap_helper (always_inline) shows
+# up under both --mode=interp (the default for the .wasm path) and
+# --mode=aot (the third invocation in symbolicate.sh).
+
+set -euo pipefail
+
+SCRIPT_DIR=$(cd "$(dirname "$0")" && pwd)
+BUILD_DIR="${SCRIPT_DIR}/build"
+
+if [ ! -x "${BUILD_DIR}/iwasm" ]; then
+    echo "iwasm not found at ${BUILD_DIR}/iwasm; build the sample first" >&2
+    exit 1
+fi
+
+cd "${BUILD_DIR}"
+./iwasm wasm-apps/trap.wasm 2>&1 | grep "^#" > call_stack.txt || true
+./iwasm wasm-apps/trap.aot 2>&1 | grep "^#" > call_stack_aot.txt || true
+
+OUT=$(mktemp)
+trap 'rm -f "$OUT"' EXIT
+"${SCRIPT_DIR}/symbolicate.sh" 2>&1 | tee "$OUT" > /dev/null
+
+assert() {
+    local pattern="$1"
+    local where="$2"
+    if ! grep -q "$pattern" "$OUT"; then
+        echo "FAIL: pattern '$pattern' ($where) not found in symbolicate.sh output" >&2
+        cat "$OUT" >&2
+        exit 1
+    fi
+}
+
+# assert_re: like assert but with POSIX extended regex (anchored line match).
+assert_re() {
+    local pattern="$1"
+    local where="$2"
+    if ! grep -Eq "$pattern" "$OUT"; then
+        echo "FAIL: regex '$pattern' ($where) did not match symbolicate.sh output" >&2
+        cat "$OUT" >&2
+        exit 1
+    fi
+}
+
+# trap.c's runtime call chain is _start -> ... -> main -> a -> b -> c
+# (with c calling __builtin_trap inside trap_helper, an always_inline
+# helper). Symbolicate.sh runs three invocations against the same call
+# stack. We assert the precise frames each one produces.
+#
+# All invocations should symbolicate the user-code frames `c`, `b`, `a`,
+# `main` (each on its own line, prefixed by the frame index).
+assert_re '^[0-9]+: c$'    "user frame: c"
+assert_re '^[0-9]+: b$'    "user frame: b"
+assert_re '^[0-9]+: a$'    "user frame: a"
+assert_re '^[0-9]+: main$' "user frame: main"
+assert "trap.c" "source file"
+
+# Inline expansion of trap_helper (always_inline) only surfaces in the
+# address-based runs (default --mode=interp and --mode=aot). It MUST
+# appear at least once in the captured output, with the
+# "(inlined into c)" annotation that print_frames adds for inlined
+# frames.
+assert_re '^[[:space:]]*0:[[:space:]]+trap_helper[[:space:]]+\(inlined into c\)$' \
+    "inline expansion: trap_helper (inlined into c)"
+
+# The AOT block is the third (last) invocation in symbolicate.sh.
+# Confirm --mode=aot also produces inline expansion (the outermost
+# frame at index 0 should still be trap_helper inlined into c).
+if ! tail -25 "$OUT" | grep -Eq '^[[:space:]]*0:[[:space:]]+trap_helper[[:space:]]+\(inlined into c\)$'; then
+    echo "FAIL: AOT block (last 25 lines) missing 'trap_helper (inlined into c)' — --mode=aot may be broken" >&2
+    cat "$OUT" >&2
+    exit 1
+fi
+
+echo "PASS [debug-tools symbolication: precise frames + inline expansion verified for both wasm and aot]"

samples/debug-tools/wasm-apps/CMakeLists.txt

@@ -34,6 +34,9 @@ function (compile_sample SOURCE_FILE)
     COMMAND ${WAMRC_BIN} --size-level=0 --enable-dump-call-stack -o ${AOT_FILE} ${WASM_FILE}
     WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}
   )
+  # Force the .wasm producer to run before the AOT step under `make -j`;
+  # the DEPENDS line above only names the file, not the CMake target.
+  add_dependencies(${FILE_NAME}_aot ${FILE_NAME})
 
   ## wasm + sourcemap
   if (DEFINED EMSCRIPTEN)

samples/debug-tools/wasm-apps/trap.c

@@ -1,7 +1,18 @@
+__attribute__((always_inline)) static inline int
+trap_helper(int n)
+{
+    /* Forced-inline helper to demonstrate inline call stack expansion.
+       The trap is inside this helper so the runtime offset falls within
+       the inlined region — addr2line.py will then show both trap_helper
+       and c() in the call chain (innermost first). */
+    __builtin_trap();
+    return n + 100;
+}
+
 int
 c(int n)
 {
-    __builtin_trap();
+    return trap_helper(n);
 }
 
 int