From 535e28db73d4e1740501a9ee0db56219a56ade02 Mon Sep 17 00:00:00 2001 From: TianlongLiang Date: Thu, 25 Jun 2026 10:15:22 +0800 Subject: [PATCH 1/2] addr2line: collapse modern/legacy resolvers into always-on interval-table overlay MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Refactor the wasm symbolicator to make its DWARF workaround explicit and unconditional. addr2line.py runs one llvm-symbolizer -f -i call per address, then overlays the outermost frame's function name from a startup-built DWARF interval table. The overlay exists because llvm-symbolizer returns the wrong outermost function name on wasm targets for two independent, source-verified reasons: - Symbol-table override on the outermost frame (LLVM design, not a bug). symbolizeInlinedCode in llvm/lib/DebugInfo/Symbolize/ SymbolizableObjectFile.cpp unconditionally rewrites the outermost frame's FunctionName from getNameFromSymbolTable when FunctionNameKind::LinkageName and UseSymbolTable are both set, which is the default. Inline frames come from getInliningInfoForAddress (a different code path) and are provably immune. - AddrDieMap invariant violation from wasm-opt -Oz -g DCE ghosts. LocationUpdater::getNewFuncStart in binaryen's src/wasm/wasm-debug.cpp returns 0 as a tombstone when an old address doesn't map to a surviving IR node, and updateDIE writes it into DW_AT_low_pc without removing the DW_TAG_subprogram. LLVM's updateAddressDieMap in DWARFUnit.cpp requires child ranges ⊆ parent ranges; these ghosts break that invariant so the map's upper_bound − 1 lookup returns an insertion-order-dependent DIE. Both mechanisms produce the same "wrong outermost frame name" symptom and both apply on any supported wasi-sdk, so the overlay runs on every resolution. build_subprogram_intervals filters three DIE classes at parse time: DW_AT_declaration=true (forward decls), low_pc == 0 (DWARF Code-section-relative zero is the function-count LEB, not a real body — this is the DCE-ghost pattern), and high_pc <= low_pc (degenerate ranges). The innermost covering DW_TAG_subprogram wins. Also includes accumulated tool improvements: - --mode flag {interp,aot,fast-interp} for different runtime offset conventions (interp post-advance, aot at-instruction-start, fast-interp's transformed in-memory bytecode) - Inline frame annotation "(inlined into )" for clarity - llvm-symbolizer preferred over llvm-addr2line for column info - Fallback for offset=0 (trap at function entry; frame_ip not captured) - Last-resort function-index name fallback when DWARF lacks PC ranges Full design rationale, source citations, and empirical truth table in test-tools/addr2line/README.md. --- test-tools/addr2line/README.md | 168 +++++++ test-tools/addr2line/addr2line.py | 729 ++++++++++++++++++++++-------- 2 files changed, 705 insertions(+), 192 deletions(-) create mode 100644 test-tools/addr2line/README.md diff --git a/test-tools/addr2line/README.md b/test-tools/addr2line/README.md new file mode 100644 index 0000000000..0254912810 --- /dev/null +++ b/test-tools/addr2line/README.md @@ -0,0 +1,168 @@ +# `addr2line.py` — symbolicating WAMR call stacks + +`addr2line.py` turns a WAMR call-stack dump (`#00: 0x0040 - $f0` lines) back +into source-level information (function names, files, line numbers, inline +chains). It is a thin orchestrator over four LLVM/wabt tools, with workarounds +for two LLVM symbolization issues on wasm targets (one a design choice, one an +invariant violation from `wasm-opt -g`). + +The two `samples/debug-tools*/` samples are end-to-end demos of the workflow. +This README focuses on the tool itself: how it works, why each subprocess is +necessary, and why the interval-table overlay is required. + +## Quick reference + +```bash +python3 addr2line.py \ + --wasi-sdk /opt/wasi-sdk \ + --wabt /opt/wabt \ + --wasm-file /path/to/app.debug.wasm \ + call_stack.txt +``` + +| Flag | Default | Purpose | +|------|---------|---------| +| `--mode={interp,aot,fast-interp}` | `interp` | How to interpret runtime offsets — see *Three offset spaces* below. | +| `--no-addr` | off | Resolve by function name only (for very old iwasm dumps that lack offsets, or for fast-interp). | + +## What the script does, step by step + +1. **Parse each captured frame** (`#NN: 0xADDR - SYMBOL`). +2. **Convert the runtime offset to a DWARF address.** WAMR reports + file-absolute byte offsets; DWARF addresses are relative to the start of + the WASM Code section, so the script subtracts `code_section_start` (read + from `wasm-objdump -h`). +3. **Apply a mode-dependent adjustment.** See *Three offset spaces* below. +4. **Resolve the address.** `llvm-symbolizer -f -i` for line info and inline + chain, with the outermost frame's function name overlaid from a startup- + built interval table (see "Why the interval-table overlay exists" below). +5. **Demangle and render.** C++ symbols pass through `llvm-cxxfilt`; inline + frames are annotated `(inlined into )` so the chain reads top-down. + +## Three offset spaces (`--mode`) + +WAMR captures `frame_ip` differently in each execution mode, so the offset in +a captured frame means different things: + +| Mode | Offset space | Adjustment | What addr2line does | +|------|--------------|-----------|----------------------| +| `interp` (default) | File-absolute, *post*-advance — interpreter has already incremented `frame_ip` past the trapping opcode (and past LEB reads inside the handler) before the trap fires. | `offset - code_start - 1` | Full symbolication: file, line, column, inline chain. | +| `aot` | File-absolute, *instruction-start* — `wamrc` emits a `commit_ip` before every WASM operation. | `offset - code_start` | Full symbolication. | +| `fast-interp` | Function-relative, into the *transformed* in-memory bytecode (opcodes replaced with handler indices, LEBs expanded to fixed widths). The mapping back to source is destroyed by the transform. | (none) | Function-name lookup only — equivalent to `--no-addr`. One frame per input; no inline expansion (the `DW_TAG_inlined_subroutine` overlay is address-keyed and unavailable here). | + +Picking the wrong `--mode` lands the resolver inside an adjacent function and +silently produces wrong file/line. + +## Why the interval-table overlay exists + +`llvm-symbolizer -e -f -i ` is the natural way to map an address +to a function + source location, with `-i` expanding inline frames. It's +correct for line/column info and for inline frame names — but its +**outermost frame's function name** goes through two paths that can both +produce wrong answers on wasm: + +1. **Symbol-table override on the outermost frame.** `symbolizeInlinedCode` + unconditionally rewrites the outermost frame's `FunctionName` from + `getNameFromSymbolTable` whenever `FunctionNameKind::LinkageName` and + `UseSymbolTable` are both set — which is the default. Inline frames are + provably NOT rewritten (the override targets only + `getMutableFrame(getNumberOfFrames() - 1)`). On some wasi-sdk / wasm + layouts the symbol table's lookup returns the wrong function for an + address that DWARF places correctly. This is an intentional LLVM design + choice, not a bug — it exists because `-gline-tables-only` builds have + only line tables in DWARF and the symbol table gives better linkage + names in that case. The tradeoff bites on full-DWARF wasm builds. +2. **`AddrDieMap` invariant violation from `wasm-opt` DCE ghosts.** + `wasm-opt -Oz -g` leaves DIEs for dead-code-eliminated compiler-rt / + libm helpers behind with `low_pc = 0` and small `high_pc` (typically + `[0, 0)` or `[0, 6)`, sometimes up to `[0, 0x80)`). This is a + consequence of binaryen's DWARF preservation model: its address- + rewrite pass in `src/wasm/wasm-debug.cpp`'s + `LocationUpdater::getNewFuncStart` returns `0` as a tombstone when + an old address doesn't map to any surviving IR node, and `updateDIE` + writes that `0` into the `DW_AT_low_pc` — but leaves the + `DW_TAG_subprogram` in place, and sometimes leaves the original + `DW_AT_high_pc`. Binaryen's README calls this out: DWARF support is + optional, tracks address relocation rather than DIE-tree + restructuring, and is "not suitable for a fully optimized release + build." LLVM's `updateAddressDieMap` in `DWARFUnit.cpp` assumes + children are inserted after parents and each child range ⊆ + parent range; the ghost DIEs violate that invariant and the map's + `upper_bound − 1` lookup returns an insertion-order-dependent DIE. + +Empirical truth table (probing an inlined address of `trigger_oob` in a +small test module across all four toolchain combinations): + +| wasi-sdk | wasm-opt -Oz -g | symbolizer outer | mechanism | +|---|---|---|---| +| 29 (clang 21) | no | `free` ❌ | Symbol-table override | +| 29 (clang 21) | yes | `free` ❌ | Symbol-table override | +| 33 (clang 22) | no | `app_main` ✅ | Neither mechanism triggers | +| 33 (clang 22) | yes | `fmaf` ❌ | AddrDieMap invariant violated by a `[low_pc=0, 0x72)` `fmaf` ghost | + +The overlay fixes all four cases. Inner (inlined) frames are not affected +because they come from a different LLVM code path +(`getInliningInfoForAddress`), which walks `getParent()` links from the +leaf DIE and does NOT go through the symbol-table override applied to +the outermost concrete frame. + +### How the overlay works + +At startup, addr2line.py runs one `llvm-dwarfdump --debug-info` pass and +builds an in-memory list `[(low_pc, high_pc, name), ...]` covering every +real callable `DW_TAG_subprogram`. Three skip guards keep the list clean: + +- `DW_AT_declaration=true` → forward declarations (e.g. WASI imports like + `__imported_wasi_snapshot_preview1_*`) have no code, so they're excluded. +- `low_pc == 0` → structurally impossible for real code, since the wasm + binary spec guarantees the Code section begins after at least a Type + + Function section header. Any `low_pc = 0` DIE is a wasm-opt DCE ghost + or a broken producer's output. +- `high_pc <= low_pc` → empty or degenerate range. Defensive; never + observed on wasi-sdk 29 or 33. + +For each captured call-stack address, `lookup_subprogram_name` returns +the innermost (smallest range) surviving DW_TAG_subprogram covering the +address. That result overwrites the outermost frame's function name in +the symbolizer output; inner frame names are left alone. + +## Tools used by addr2line.py + +| Tool | From | Why this tool, and not another | +|------|------|--------------------------------| +| `llvm-symbolizer` | wasi-sdk | The actual address → `(function, file:line:column)` resolver, with inline expansion via `-i`. Ships in wasi-sdk 29+ (the project's minimum supported version). | +| `llvm-dwarfdump` | wasi-sdk | Single `--debug-info` pass at startup builds a `(low_pc, high_pc, name)` interval table for every real callable `DW_TAG_subprogram`. Used to overlay the outermost frame's function name in every resolution. The overlay is necessary because `llvm-symbolizer -f` unconditionally rewrites that frame's `FunctionName` from an object-file symbol-table lookup (LLVM design, not a bug) and — separately — `wasm-opt -Oz -g` output can violate the `AddrDieMap` invariant so DWARF's own DIE lookup returns the wrong entry. Reading DWARF ranges directly bypasses both mechanisms. No equivalent CLI exposes this lookup as structured output, so we parse the text. | +| `wasm-objdump` | wabt | Reports the Code-section start offset (needed to convert file-absolute runtime offsets to DWARF addresses) and the function-index → name table (a last-resort fallback for declaration-only DW_TAG_subprograms like `__main_void`). | +| `llvm-cxxfilt` | wasi-sdk | C++ symbol demangling, applied as a final pass on resolved names. | + +The script also detects an emscripten-style sourceMappingURL section and +falls back to `emsymbolizer` for that case. + +## Behavior matrix + +| Input | Resolver path | Output | +|-------|---------------|--------| +| `--mode=interp` or `--mode=aot`, address in DWARF range | Unified | `llvm-symbolizer -f -i` for line info + inline chain; outermost frame's function name overlaid from a startup-built interval table (innermost match wins) | +| `--mode=fast-interp` | Function-name | Single frame per input, no inline expansion, no source mapping | +| `--no-addr` | Function-name | Resolves the *function* — file:line refers to its declaration, not the trap site | +| Offset = 0 | Function-name fallback (with `(offset=0 — function entry)` note) | Used when WAMR couldn't capture `frame_ip` (e.g., trap at function entry, or top frame of an `iwasm -f` invocation) | +| Address outside any DW_TAG_subprogram | wasm-objdump function-index table | Single name; useful for declaration-only entries | +| Source has emscripten `sourceMappingURL` custom section | `emsymbolizer` source-map path | Single frame per address; no inline expansion (source maps don't carry it) | + +## Tests + +Run the pytest suite under `test-tools/addr2line/tests/`. It builds tiny +purpose-built C/C++ apps, captures their trap sites, and asserts the resolver +behaves correctly under a number of scenarios (single trap, always_inline, +deep inline chain, cross-TU LTO inlining, mid-function trap, multi-frame +call stack, C++ demangling, AOT mode, fast-interp mode, `--no-addr`, +`offset=0`, empty input). Multi-SDK runs (`pytest --multi-sdk`) exercise +the overlay across multiple toolchain versions and require multiple wasi-sdk +installations under `/opt`. + +## See also + +- `test-tools/addr2line/addr2line.py` — the script +- `test-tools/addr2line/tests/` — pytest harness +- `samples/debug-tools/` — minimal end-to-end sample +- `samples/debug-tools-optimized/` — production-realistic workflow (LTO, wasm-opt, strip, AOT, debug companion) diff --git a/test-tools/addr2line/addr2line.py b/test-tools/addr2line/addr2line.py index 800ba37504..810609096c 100644 --- a/test-tools/addr2line/addr2line.py +++ b/test-tools/addr2line/addr2line.py @@ -14,7 +14,7 @@ """ This is a tool to convert addresses, which are from a call-stack dump generated by iwasm, into line info for a wasm file. -When a wasm file is compiled with debug info, it is possible to transfer the address to line info. +When a wasm file is compiled with debug info (-g), it is possible to transfer the address to line info. For example, there is a call-stack dump: @@ -26,27 +26,47 @@ ``` - store the call-stack dump into a file, e.g. call_stack.txt -- run the following command to convert the address into line info: +- run the following command: ``` - $ cd test-tools/addr2line - $ python3 addr2line.py --wasi-sdk --wabt --wasm-file call_stack.txt + $ python3 addr2line.py --wasi-sdk --wabt --wasm-file call_stack.txt ``` - The script will use *wasm-objdump* in wabt to transform address, then use *llvm-dwarfdump* to lookup the line info for each address - in the call-stack dump. -- if addresses are not available in the stack trace (i.e. iwasm <= 1.3.2) or iwasm is used in fast interpreter mode, - run the following command to convert the function index into line info (passing the `--no-addr` option): - ``` - $ python3 addr2line.py --wasi-sdk --wabt --wasm-file call_stack.txt --no-addr - ``` - The script will use *wasm-objdump* in wabt to get the function names corresponding to function indexes, then use *llvm-dwarfdump* to lookup the line info for each - function index in the call-stack dump. + +The script handles two workflows: +- Same-binary: pass the wasm file that ran (must have DWARF debug info) +- Companion: pass a sibling debug companion (.debug.wasm produced by `wasm-opt -Oz -g`). + The production binary that ran can be stripped — the companion has identical code + layout but retains DWARF, so the runtime offsets map correctly. + +Inline expansion (DW_TAG_inlined_subroutine) is automatic — no flag needed. + +By default, offsets are interpreted as coming from classic interpreter mode +(frame_ip has advanced past the trapping opcode). For AOT call stacks, pass +`--mode=aot` so offsets are used verbatim — wamrc commits ip at the start of +each WASM operation, not after. + +For fast-interp call stacks, pass `--mode=fast-interp`. Fast-interp transforms +the bytecode in-memory at load time, so its offsets are meaningless for source +mapping; the script falls back to function-name lookup (same as `--no-addr`). + +If addresses are not available (iwasm <= 1.3.2), pass `--no-addr` to look up +by function index/name only. + +- addr2line.py always applies an interval-table overlay to the outermost + frame's function name. `llvm-symbolizer` unconditionally rewrites that + frame's name from an object-file symbol-table lookup (see + `symbolizeInlinedCode` in + `llvm/lib/DebugInfo/Symbolize/SymbolizableObjectFile.cpp`), and the + symbol-table can return wrong names on wasm. Separately, + `wasm-opt -Oz -g` leaves dead-code-eliminated `DW_TAG_subprogram` DIEs + with `low_pc = 0` behind, violating LLVM's `AddrDieMap` parents-before- + children invariant. The overlay bypasses both by reading DWARF ranges + directly. Inner inlined frames come from `getInliningInfoForAddress` + (a different code path) and are not affected by either mechanism. """ def locate_sourceMappingURL_section(wasm_objdump: Path, wasm_file: Path) -> bool: - """ - Figure out if the wasm file has a sourceMappingURL section. - """ + """Figure out if the wasm file has a sourceMappingURL section.""" cmd = f"{wasm_objdump} -h {wasm_file}" p = subprocess.run( shlex.split(cmd), @@ -55,24 +75,18 @@ def locate_sourceMappingURL_section(wasm_objdump: Path, wasm_file: Path) -> bool text=True, universal_newlines=True, ) - outputs = p.stdout.split(os.linesep) - - for line in outputs: - line = line.strip() - if "sourceMappingURL" in line: + for line in p.stdout.splitlines(): + if "sourceMappingURL" in line.strip(): return True - return False def get_code_section_start(wasm_objdump: Path, wasm_file: Path) -> int: """ - Find the start offset of Code section in a wasm file. + Find the start file offset of the Code section in a wasm file. - if the code section header likes: + Code section header looks like: Code start=0x0000017c end=0x00004382 (size=0x00004206) count: 47 - - the start offset is 0x0000017c """ cmd = f"{wasm_objdump} -h {wasm_file}" p = subprocess.run( @@ -82,155 +96,228 @@ def get_code_section_start(wasm_objdump: Path, wasm_file: Path) -> int: text=True, universal_newlines=True, ) - outputs = p.stdout.split(os.linesep) - - for line in outputs: + for line in p.stdout.splitlines(): line = line.strip() - if "Code" in line: - return int(line.split()[1].split("=")[1], 16) - + if "Code" in line and "start=" in line: + m = re.search(r"start=(0x[0-9a-fA-F]+)", line) + if m: + return int(m.group(1), 16) return -1 -def get_line_info_from_function_addr_dwarf( - dwarf_dump: Path, wasm_file: Path, offset: int -) -> tuple[str, str, str, str]: +def build_subprogram_intervals(dwarf_dump: Path, wasm_file: Path) -> list: """ - Find the location info of a given offset in a wasm file. + Return a list of (low_pc, high_pc, name) for every real callable + DW_TAG_subprogram in the wasm's DWARF. + + Filtered out during parse: + - DIEs with DW_AT_declaration=true (forward declarations; no code). + - DIEs with low_pc == 0. DWARF low_pc is Code-section-relative, and + the section's first byte is the function-count LEB, not any real + function body — so any low_pc=0 DIE is either a declaration or a + dead-code ghost left by wasm-opt (see the "binaryen tombstone" + note in the design doc). + - DIEs with high_pc <= low_pc (empty or degenerate ranges). + + The resulting list is used to overlay the outermost frame's function + name — llvm-symbolizer picks the wrong DW_TAG_subprogram at addresses + where multiple DIEs claim to cover the same PC. See the design doc at + docs/superpowers/specs/2026-07-06-addr2line-collapse-modern-legacy-design.md + for the two independent bug sources. + + Parsed once per run; lookups against the result are O(N) in the + subprogram count, which is small. """ - cmd = f"{dwarf_dump} --lookup={offset} {wasm_file}" - p = subprocess.run( + cmd = f"{dwarf_dump} --debug-info {wasm_file}" + raw = subprocess.run( shlex.split(cmd), check=False, capture_output=True, text=True, universal_newlines=True, - ) - outputs = p.stdout.split(os.linesep) - - function_name, function_file = "", "unknown" - function_line, function_column = "?", "?" - - for line in outputs: - line = line.strip() - - if "DW_AT_name" in line: - function_name = get_dwarf_tag_value("DW_AT_name", line) - - if "DW_AT_decl_file" in line: - function_file = get_dwarf_tag_value("DW_AT_decl_file", line) - - if "Line info" in line: - _, function_line, function_column = parse_line_info(line) - - return (function_name, function_file, function_line, function_column) + ).stdout + + intervals = [] + sp_indent = -1 + low = high = name = None + is_declaration = False + + def _flush(): + # Skip guards: declaration-only, low_pc=0 (wasm-opt DCE ghost or + # invalid), or empty/degenerate range. + if is_declaration: + return + if low is None or low == 0: + return + if high is None or high <= low: + return + if name is None: + return + intervals.append((low, high, name)) + + for raw_line in raw.splitlines(): + # llvm-dwarfdump prefixes DIE-tag lines with their offset + # ("0xNNN:") whose width varies; replace with a fixed sentinel so + # the tag-line indent matches attribute-line indent. + body = re.sub(r"^0x[0-9a-fA-F]+:", " " * 11, raw_line) + stripped = body.lstrip() + if not stripped: + continue + indent = len(body) - len(stripped) + if stripped.startswith("DW_TAG_subprogram"): + _flush() + sp_indent = indent + low = high = name = None + is_declaration = False + continue + if sp_indent < 0: + continue + # A DIE at sp_indent or shallower closes the current subprogram. + if stripped.startswith("DW_TAG_") and indent <= sp_indent: + _flush() + sp_indent = -1 + low = high = name = None + is_declaration = False + continue + # Only attributes one level deeper than the tag belong to the + # subprogram itself; deeper indents are nested DIEs. + if indent != sp_indent + 2: + continue -def get_dwarf_tag_value(tag: str, line: str) -> str: - # Try extracting value as string - STR_PATTERN = rf"{tag}\s+\(\"(.*)\"\)" - m = re.match(STR_PATTERN, line) - if m: - return m.groups()[0] + m = re.match(r"DW_AT_low_pc\s+\(0x([0-9a-fA-F]+)\)", stripped) + if m: + low = int(m.group(1), 16) + continue + m = re.match(r"DW_AT_high_pc\s+\(0x([0-9a-fA-F]+)\)", stripped) + if m: + v = int(m.group(1), 16) + # high_pc is rendered as either an absolute address or — for + # DW_FORM_data* — as size-from-low_pc; pick the larger of the + # two interpretations. + high = v if low is None or v >= low else low + v + continue + m = re.match(r'DW_AT_name\s+\("(.+)"\)', stripped) + if m and name is None: + name = m.group(1) + continue + # DW_AT_declaration(true) → forward declaration; never callable. + if stripped.startswith("DW_AT_declaration") and "true" in stripped: + is_declaration = True - # Try extracting value as integer - INT_PATTERN = rf"{tag}\s+\((\d+)\)" - m = re.match(INT_PATTERN, line) - return m.groups()[0] + _flush() + return intervals -def get_line_info_from_function_name_dwarf( - dwarf_dump: Path, wasm_file: Path, function_name: str -) -> tuple[str, str, str]: - """ - Find the location info of a given function in a wasm file. +def lookup_subprogram_name(intervals: list, dwarf_addr: int) -> str: """ - cmd = f"{dwarf_dump} --name={function_name} {wasm_file}" - p = subprocess.run( - shlex.split(cmd), - check=False, - capture_output=True, - text=True, - universal_newlines=True, - ) - outputs = p.stdout.split(os.linesep) - - function_name, function_file = "", "unknown" - function_line = "?" - - for line in outputs: - line = line.strip() - - if "DW_AT_name" in line: - function_name = get_dwarf_tag_value("DW_AT_name", line) + Return the name of the innermost (smallest-range) DW_TAG_subprogram + whose [low_pc, high_pc) covers `dwarf_addr`, or None. - if "DW_AT_decl_file" in line: - function_file = get_dwarf_tag_value("DW_AT_decl_file", line) - - if "DW_AT_decl_line" in line: - function_line = get_dwarf_tag_value("DW_AT_decl_line", line) - - return (function_name, function_file, function_line) - - -def get_line_info_from_function_addr_sourcemapping( - emsymbolizer: Path, wasm_file: Path, offset: int -) -> tuple[str, str, str, str]: + "Innermost" matters because wasi-libc declarations (low_pc=0, + high_pc spanning much of the binary) overlap real functions; we want + the tightly-bounded one. """ - Find the location info of a given offset in a wasm file which is compiled with emcc. - - {emsymbolizer} {wasm_file} {offset of file} - - there usually are two lines: - ?? - relative path to source file:line:column + best = None + for low, high, name in intervals: + if low <= dwarf_addr < high: + if best is None or (high - low) < (best[1] - best[0]): + best = (low, high, name) + return best[2] if best else None + + +def resolve_address( + symbolizer: Path, wasm_file: Path, dwarf_addr: int, + subprogram_intervals: list, +) -> list: """ - debug_info_source = wasm_file.with_name(f"{wasm_file.name}.map") - cmd = f"{emsymbolizer} -t code -f {debug_info_source} {wasm_file} {offset}" + Resolve one DWARF address to an inline-frame chain, with an + outermost-name overlay from the interval table. + + Runs: -e -f -i 0x + + The symbolizer produces pairs of lines per frame (function name, + then file:line[:column]). With -i, multiple pairs are returned — + innermost inline frame first, outermost real function last. + + The symbolizer's outermost function-name lookup is unreliable on + wasm targets for two independent reasons: + 1. Symbol-table override (LLVM design, not a bug). + `symbolizeInlinedCode` unconditionally rewrites the outermost + frame's FunctionName from `getNameFromSymbolTable` whenever + `FunctionNameKind::LinkageName` and `UseSymbolTable` are both + set — which is the default. Inline frames are provably NOT + rewritten (the override targets only + `getMutableFrame(getNumberOfFrames() - 1)`), so an inline + chain like `do_bad_access → trigger_oob → app_main` from + DWARF gets its LAST entry overwritten by whatever the symbol + table's `getNameFromSymbolTable` returns for that address. + On some wasi-sdk / wasm layouts that returns a wrong function. + 2. `wasm-opt -Oz -g` leaves DIEs for dead-code-eliminated + compiler-rt / libm helpers behind with low_pc=0 and small + high_pc. Those ghost DIEs violate LLVM `AddrDieMap`'s + parents-before-children invariant (child range ⊆ parent + range), so the map's `upper_bound − 1` lookup returns a + ghost DIE for real addresses. This is separate from + mechanism 1 and applies on any clang version. + In both cases only the OUTERMOST frame's function name is wrong; + inline frames and line/column info are correct. See the + design doc at + docs/superpowers/specs/2026-07-06-addr2line-collapse-modern-legacy-design.md + for source-verified references into + llvm/lib/DebugInfo/Symbolize/SymbolizableObjectFile.cpp and + llvm/lib/DebugInfo/DWARF/DWARFUnit.cpp. + + This function unconditionally overlays the outermost frame's + function name with the result of an interval-table lookup that + picks the innermost DW_TAG_subprogram covering `dwarf_addr`. When + the interval table has no covering entry, the symbolizer's + outermost name is left as-is. + + Returns a list of (function_name, file, line, column) tuples. + Each field is a string; line and column may be "?" if unknown. + """ + addr_str = f"0x{dwarf_addr:x}" + cmd = f"{symbolizer} -e {wasm_file} -f -i {addr_str}" p = subprocess.run( shlex.split(cmd), check=False, capture_output=True, text=True, universal_newlines=True, - cwd=Path.cwd(), ) - outputs = p.stdout.split(os.linesep) - - function_name, function_file = "", "unknown" - function_line, function_column = "?", "?" + output_lines = [line for line in p.stdout.splitlines() if line.strip()] - for line in outputs: - line = line.strip() - - if not line: - continue + frames = [] + for i in range(0, len(output_lines) - 1, 2): + func_name = output_lines[i].strip() + location = output_lines[i + 1].strip() - m = re.match(r"(.*):(\d+):(\d+)", line) + # Parse "file:line:column" or "file:line" or "??:0" + m = re.match(r"^(.*):(\d+):(\d+)$", location) if m: - function_file, function_line, function_column = m.groups() - continue + file, line, column = m.group(1), m.group(2), m.group(3) else: - # it's always ??, not sure about that - if "??" != line: - function_name = line + m = re.match(r"^(.*):(\d+)$", location) + if m: + file, line, column = m.group(1), m.group(2), "?" + else: + file, line, column = location, "?", "?" - return (function_name, function_file, function_line, function_column) + frames.append((func_name, file, line, column)) + # Overlay the outermost frame's name from the interval table when + # available. Inner frames are left alone. + if frames: + name = lookup_subprogram_name(subprogram_intervals, dwarf_addr) + if name: + _, file, line, column = frames[-1] + frames[-1] = (name, file, line, column) -def parse_line_info(line_info: str) -> tuple[str, str, str]: - """ - line_info -> [file, line, column] - """ - PATTERN = r"Line info: file \'(.+)\', line ([0-9]+), column ([0-9]+)" - m = re.search(PATTERN, line_info) - assert m is not None + return frames - file, line, column = m.groups() - return (file, int(line), int(column)) - -def parse_call_stack_line(line: str) -> tuple[str, str, str]: +def parse_call_stack_line(line: str) -> tuple: """ New format (WAMR > 1.3.2): #00: 0x0a04 - $f18 => (00, 0x0a04, $f18) @@ -241,7 +328,6 @@ def parse_call_stack_line(line: str) -> tuple[str, str, str]: _start (always): #05: 0x011f - _start => (05, 0x011f, _start) """ - # New format and Text format and _start PATTERN = r"#([0-9]+): 0x([0-9a-f]+) - (\S+)" m = re.match(PATTERN, line) @@ -257,7 +343,8 @@ def parse_call_stack_line(line: str) -> tuple[str, str, str]: return None -def parse_module_functions(wasm_objdump: Path, wasm_file: Path) -> dict[str, str]: +def parse_module_functions(wasm_objdump: Path, wasm_file: Path) -> dict: + """Map function index to name from wasm-objdump output.""" function_index_to_name = {} cmd = f"{wasm_objdump} -x {wasm_file} --section=function" @@ -268,16 +355,13 @@ def parse_module_functions(wasm_objdump: Path, wasm_file: Path) -> dict[str, str text=True, universal_newlines=True, ) - outputs = p.stdout.split(os.linesep) - - for line in outputs: - if not f"func[" in line: + for line in p.stdout.splitlines(): + if "func[" not in line: continue - PATTERN = r".*func\[([0-9]+)\].*<(.*)>" m = re.match(PATTERN, line) - assert m is not None - + if m is None: + continue index = m.groups()[0] name = m.groups()[1] function_index_to_name[index] = name @@ -285,6 +369,121 @@ def parse_module_functions(wasm_objdump: Path, wasm_file: Path) -> dict[str, str return function_index_to_name +def get_line_info_from_function_name_dwarf( + dwarf_dump: Path, wasm_file: Path, function_name: str +) -> tuple: + """ + Used by --no-addr mode (fast-interp call stacks, or WAMR <= 1.3.2 + output where addresses are absent). + + Resolves a function name to its declaration's (file, decl_line) by + parsing `llvm-dwarfdump --name=` output: collect every + matching DW_TAG_subprogram block, then prefer a non-sysroot match + over a wasi-libc declaration of the same name (short names like + `b`, `c` frequently collide with wasi-libc helpers). + + TODO: limitation — `llvm-symbolizer` is address-driven and has no + name-to-source mode, so this helper manually parses llvm-dwarfdump + text output. The resolved location is the function's declaration + line, not the call site. A robust replacement would need a + programmatic DWARF reader (e.g. pyelftools) or a future LLVM CLI + exposing name-keyed lookup. + + Returns (function_name, file, line). + """ + cmd = f"{dwarf_dump} --name={function_name} {wasm_file}" + p = subprocess.run( + shlex.split(cmd), + check=False, + capture_output=True, + text=True, + universal_newlines=True, + ) + + # Collect every (name, file, line) triple from the output. Each + # DW_TAG_subprogram block is separated by a blank line. + candidates = [] + cur = {} + for line in p.stdout.splitlines() + [""]: + line = line.strip() + if not line: + if cur: + candidates.append(( + cur.get("name", ""), + cur.get("file", "unknown"), + cur.get("line", "?"), + )) + cur = {} + continue + m = re.match(r'DW_AT_name\s+\("(.+)"\)', line) + if m: + cur["name"] = m.group(1) + continue + m = re.match(r'DW_AT_decl_file\s+\("(.+)"\)', line) + if m: + cur["file"] = m.group(1) + continue + m = re.match(r"DW_AT_decl_line\s+\((\d+)\)", line) + if m: + cur["line"] = m.group(1) + + # Prefer a candidate whose source file isn't from wasi-sysroot / + # wasi-libc / wasi-sdk — short names like `b`, `c` collide with + # internal libc helpers and llvm-dwarfdump's name lookup returns + # every match across the binary. + def _is_sysroot(p): + return "wasi-sysroot" in p or "wasi-libc" in p or p.startswith("wasisdk://") + + for name, file, line in candidates: + if name == function_name and not _is_sysroot(file): + return (name, file, line) + # Fall back to the last candidate (preserves prior behavior when + # all matches are sysroot-ish or the name doesn't match exactly). + if candidates: + return candidates[-1] + return ("", "unknown", "?") + + +def get_line_info_from_function_addr_sourcemapping( + emsymbolizer: Path, wasm_file: Path, offset: int +) -> tuple: + """ + Find the location info of a given offset in a wasm file compiled with emcc. + + {emsymbolizer} {wasm_file} {offset} + + Output is two lines: + ?? + relative path to source:line:column + """ + debug_info_source = wasm_file.with_name(f"{wasm_file.name}.map") + cmd = f"{emsymbolizer} -t code -f {debug_info_source} {wasm_file} {offset}" + p = subprocess.run( + shlex.split(cmd), + check=False, + capture_output=True, + text=True, + universal_newlines=True, + cwd=Path.cwd(), + ) + + function_name, function_file = "", "unknown" + function_line, function_column = "?", "?" + + for line in p.stdout.splitlines(): + line = line.strip() + if not line: + continue + m = re.match(r"(.*):(\d+):(\d+)", line) + if m: + function_file, function_line, function_column = m.groups() + continue + if "??" != line: + function_name = line + + return (function_name, function_file, function_line, function_column) + + def demangle(cxxfilt: Path, function_name: str) -> str: cmd = f"{cxxfilt} -n {function_name}" p = subprocess.run( @@ -297,45 +496,133 @@ def demangle(cxxfilt: Path, function_name: str) -> str: return p.stdout.strip() +def print_frames(index_str: str, frames: list, cxxfilt: Path) -> None: + """ + Print resolved frames with consistent indentation. Inlined functions + (frames before the outermost) get an "(inlined into )" suffix + so the chain relationship is unambiguous. + + Single frame (no inlines): + 0: c + at trap.c:4:5 + + Multiple frames (inline chain — innermost first): + 0: do_bad_access (inlined into trigger_oob) + at oob_access.c:11:5 + trigger_oob (inlined into app_main) + at oob_main.c:17:5 + app_main + at oob_main.c:23:5 + + The first frame uses ":" prefix; subsequent frames use the same + width of leading spaces for vertical alignment. + """ + prefix = f"{index_str}:" + indent = " " * len(prefix) + + demangled_names = [ + demangle(cxxfilt, fn) if fn != "??" else "" + for fn, _, _, _ in frames + ] + + for i, (func_name, file, line, column) in enumerate(frames): + leading = prefix if i == 0 else indent + # All frames except the outermost are inlined into the next one in the chain. + if i < len(frames) - 1: + suffix = f" (inlined into {demangled_names[i + 1]})" + else: + suffix = "" + print(f"{leading} {demangled_names[i]}{suffix}") + # Don't print "??:0:0" — collapse to "??:0". + if file == "??" and line in ("0", "?"): + print("\tat ??:0") + elif column != "?": + print(f"\tat {file}:{line}:{column}") + else: + print(f"\tat {file}:{line}") + + def main(): parser = argparse.ArgumentParser(description="addr2line for wasm") - parser.add_argument("--wasi-sdk", type=Path, help="path to wasi-sdk") - parser.add_argument("--wabt", type=Path, help="path to wabt") - parser.add_argument("--wasm-file", type=Path, help="path to wasm file") + parser.add_argument("--wasi-sdk", type=Path, required=True, help="path to wasi-sdk") + parser.add_argument("--wabt", type=Path, required=True, help="path to wabt") + parser.add_argument("--wasm-file", type=Path, required=True, help="path to wasm file (must have DWARF)") parser.add_argument("call_stack_file", type=Path, help="path to a call stack file") parser.add_argument( "--no-addr", action="store_true", help="use call stack without addresses or from fast interpreter mode", ) + parser.add_argument( + "--mode", + choices=["interp", "aot", "fast-interp"], + default="interp", + help=( + "WAMR execution mode that produced the call stack. " + "interp: classic interpreter, frame_ip is post-advance, subtract 1 " + "from offsets to land on the trapping opcode. " + "aot: wamrc commits ip at the start of each WASM operation, use " + "offsets verbatim. " + "fast-interp: offsets are relative to a transformed in-memory " + "bytecode and cannot be mapped to source line — falls back to " + "function-name lookup (same as --no-addr). " + "Default: interp." + ), + ) parser.add_argument("--emsdk", type=Path, help="path to emsdk") args = parser.parse_args() + # fast-interp offsets aren't mappable; treat as --no-addr + if args.mode == "fast-interp": + args.no_addr = True + wasm_objdump = args.wabt.joinpath("bin/wasm-objdump") - assert wasm_objdump.exists() + assert wasm_objdump.exists(), f"wasm-objdump not found at {wasm_objdump}" + + # llvm-symbolizer handles address->source resolution and inline-frame + # expansion via `-f -i`. Ships in wasi-sdk 29+, which is the project's + # minimum supported wasi-sdk for the debug-tools samples. + llvm_symbolizer = args.wasi_sdk.joinpath("bin/llvm-symbolizer") + assert llvm_symbolizer.exists(), ( + f"llvm-symbolizer not found at {llvm_symbolizer}; " + f"need wasi-sdk 29 or newer" + ) llvm_dwarf_dump = args.wasi_sdk.joinpath("bin/llvm-dwarfdump") - assert llvm_dwarf_dump.exists() + assert llvm_dwarf_dump.exists(), f"llvm-dwarfdump not found at {llvm_dwarf_dump}" llvm_cxxfilt = args.wasi_sdk.joinpath("bin/llvm-cxxfilt") - assert llvm_cxxfilt.exists() + assert llvm_cxxfilt.exists(), f"llvm-cxxfilt not found at {llvm_cxxfilt}" emcc_production = locate_sourceMappingURL_section(wasm_objdump, args.wasm_file) if emcc_production: if args.emsdk is None: print("Please provide the path to emsdk via --emsdk") return -1 - emsymbolizer = args.emsdk.joinpath("upstream/emscripten/emsymbolizer") assert emsymbolizer.exists() code_section_start = get_code_section_start(wasm_objdump, args.wasm_file) if code_section_start == -1: + print("Could not find Code section in wasm file", file=sys.stderr) return -1 function_index_to_name = parse_module_functions(wasm_objdump, args.wasm_file) - assert args.call_stack_file.exists() + # Build the DW_TAG_subprogram interval table once at startup. The + # per-frame resolve_address() call overlays the outermost function + # name from this table to fix LLVM's mis-reports on wasm: the + # symbolizer applies an unconditional symbol-table override to the + # outermost frame (see SymbolizableObjectFile.cpp), and wasm-opt + # DCE ghosts violate AddrDieMap's invariant (see DWARFUnit.cpp). + # Empty when we're on the emscripten sourceMappingURL path, which + # doesn't use DWARF. + subprogram_intervals = ( + [] if emcc_production + else build_subprogram_intervals(llvm_dwarf_dump, args.wasm_file) + ) + + assert args.call_stack_file.exists(), f"call stack file not found: {args.call_stack_file}" with open(args.call_stack_file, "rt", encoding="ascii") as f: for i, line in enumerate(f): line = line.strip() @@ -348,63 +635,121 @@ def main(): continue _, offset, index = splitted + index_str = str(i) + if args.no_addr: # FIXME: w/ emcc production if not index.startswith("$f"): # E.g. _start or Text format - print(f"{i}: {index}") + print(f"{index_str}: {index}") continue - index = index[2:] + func_idx = index[2:] - if index not in function_index_to_name: - print(f"{i}: {line}") + if func_idx not in function_index_to_name: + print(f"{index_str}: {line}") continue if not emcc_production: _, function_file, function_line = ( get_line_info_from_function_name_dwarf( - llvm_dwarf_dump, - args.wasm_file, - function_index_to_name[index], + llvm_dwarf_dump, args.wasm_file, + function_index_to_name[func_idx], ) ) else: - _, function_file, function_line = _, "unknown", "?" + function_file, function_line = "unknown", "?" - function_name = demangle(llvm_cxxfilt, function_index_to_name[index]) - print(f"{i}: {function_name}") + function_name = demangle(llvm_cxxfilt, function_index_to_name[func_idx]) + print(f"{index_str}: {function_name}") print(f"\tat {function_file}:{function_line}") - else: - offset = int(offset, 16) - # match the algorithm in wasm_interp_create_call_stack() - # either a *offset* to *code* section start - # or a *offset* in a file - assert offset > code_section_start - offset = offset - code_section_start - - if emcc_production: - function_name, function_file, function_line, function_column = ( - get_line_info_from_function_addr_sourcemapping( - emsymbolizer, args.wasm_file, offset + continue + + # Address-based lookup + offset_int = int(offset, 16) + + # WAMR reports offset 0 when frame_ip wasn't captured (e.g., trap + # at function entry, or top frame of an iwasm -f invocation where + # ip wasn't sync'd before the trap). Fall back to function-name + # lookup for the function header instead of asserting. + if offset_int == 0 or offset_int <= code_section_start: + if not emcc_production: + # Resolve function name from index ($fN) or use the name directly + if index.startswith("$f"): + func_idx = index[2:] + func_name = function_index_to_name.get(func_idx, index) + else: + func_name = index + _, function_file, function_line = ( + get_line_info_from_function_name_dwarf( + llvm_dwarf_dump, args.wasm_file, func_name ) ) - else: - function_name, function_file, function_line, function_column = ( - get_line_info_from_function_addr_dwarf( - llvm_dwarf_dump, args.wasm_file, offset - ) + print(f"{index_str}: {demangle(llvm_cxxfilt, func_name)}") + print(f"\tat {function_file}:{function_line} " + f"(offset=0 — function entry, no inline info)") + continue + # emcc path: just print raw + print(f"{index_str}: {index}") + print(f"\tat unknown:?:? (offset=0 — frame_ip not captured)") + continue + + # Mode-dependent offset adjustment: + # - interp: subtract 1 because frame_ip has advanced past the trapping + # opcode before the trap handler runs (FETCH_OPCODE_AND_DISPATCH + # increments ip before dispatching, plus LEB reads inside handlers). + # - aot: use offset verbatim. wamrc commits ip at the start of each + # WASM operation, before any LEB reads — the offset is already at + # the opcode boundary. + # (fast-interp is handled above by forcing --no-addr; we never reach + # this branch for it.) + adjustment = 1 if args.mode == "interp" else 0 + dwarf_addr = offset_int - code_section_start - adjustment + + if emcc_production: + # Source-map path doesn't support inline expansion; emit single frame. + func_name, file, line, column = ( + get_line_info_from_function_addr_sourcemapping( + emsymbolizer, args.wasm_file, dwarf_addr ) + ) + # Fall back to function index name if unresolved + if func_name == "" and index.startswith("$f"): + func_name = function_index_to_name.get(index[2:], index) + func_name = demangle(llvm_cxxfilt, func_name) + print(f"{index_str}: {func_name}") + print(f"\tat {file}:{line}:{column}") + continue - # if can't parse function_name, use name section or - if function_name == "": - if index.startswith("$f"): - function_name = function_index_to_name.get(index[2:], index) - else: - function_name = index + frames = resolve_address( + llvm_symbolizer, args.wasm_file, dwarf_addr, + subprogram_intervals, + ) - function_name = demangle(llvm_cxxfilt, function_name) + if not frames: + # Fall back to function index name + if index.startswith("$f"): + func_name = function_index_to_name.get(index[2:], index) + else: + func_name = index + print(f"{index_str}: {demangle(llvm_cxxfilt, func_name)}") + print(f"\tat unknown:?:?") + continue - print(f"{i}: {function_name}") - print(f"\tat {function_file}:{function_line}:{function_column}") + # Last-resort fallback for the outermost frame: when neither + # llvm-symbolizer nor the dwarfdump subprogram-name overlay + # produced a clean name (e.g. the address is in a function + # whose DWARF has no PC range — declaration-only + # DW_TAG_subprogram), substitute the function index -> name + # from wasm-objdump. + if index.startswith("$f"): + expected = function_index_to_name.get(index[2:]) + actual = frames[-1][0] + if expected and ( + actual == "??" + or (actual != expected and frames[-1][1] in ("??", "unknown")) + ): + frames[-1] = (expected,) + frames[-1][1:] + + print_frames(index_str, frames, llvm_cxxfilt) return 0 From c2ecb3e9189860ca9bd8d3d9aa208f9ee4b72397 Mon Sep 17 00:00:00 2001 From: TianlongLiang Date: Thu, 25 Jun 2026 10:15:42 +0800 Subject: [PATCH 2/2] samples(debug-tools): refresh existing sample for the addr2line refactor - README: document the --mode flag and inline expansion behavior, and point to the tool README for the interval-table overlay rationale. - symbolicate.sh: pass --mode aot for the AOT call-stack invocation. - verify.sh (new): assertion-based smoke test that runs the existing symbolicate.sh and grep-checks for inline expansion (trap_helper) in both the wasm and AOT outputs. Replaces the inline assertions the CI step previously carried. - wasm-apps/trap.c: add a trap_helper marked __attribute__((always_inline)) that the trap site falls inside, so addr2line.py can demonstrate inline-frame annotation against this sample. Depends on the addr2line.py refactor (--mode flag). --- samples/debug-tools/README.md | 97 ++++++++++++++++++-- samples/debug-tools/symbolicate.sh | 73 ++++++++++----- samples/debug-tools/verify.sh | 83 +++++++++++++++++ samples/debug-tools/wasm-apps/CMakeLists.txt | 3 + samples/debug-tools/wasm-apps/trap.c | 13 ++- 5 files changed, 237 insertions(+), 32 deletions(-) mode change 100644 => 100755 samples/debug-tools/symbolicate.sh create mode 100755 samples/debug-tools/verify.sh diff --git a/samples/debug-tools/README.md b/samples/debug-tools/README.md index b0358b9e4f..7ed69b0882 100644 --- a/samples/debug-tools/README.md +++ b/samples/debug-tools/README.md @@ -52,25 +52,104 @@ $ python3 ../../../test-tools/addr2line/addr2line.py \ call_stack.txt ``` +This sample also ships `symbolicate.sh` (runs the three modes shown below in +sequence) and `verify.sh` (asserts inline expansion appears in both the +classic-interp and AOT outputs — used by CI, useful locally as a smoke test). +Both expect to be run from the sample root after building: + +```bash +$ ./symbolicate.sh +$ ./verify.sh +``` + The output should be something like: ```text -0: c - at wasm-micro-runtime/samples/debug-tools/wasm-apps/trap.c:5:1 +0: trap_helper (inlined into c) + at /path/to/wasm-micro-runtime/samples/debug-tools/wasm-apps/trap.c:9:5 + c + at /path/to/wasm-micro-runtime/samples/debug-tools/wasm-apps/trap.c:16:12 1: b - at wasm-micro-runtime/samples/debug-tools/wasm-apps/trap.c:11:12 + at /path/to/wasm-micro-runtime/samples/debug-tools/wasm-apps/trap.c:23:12 2: a - at wasm-micro-runtime/samples/debug-tools/wasm-apps/trap.c:17:12 + at /path/to/wasm-micro-runtime/samples/debug-tools/wasm-apps/trap.c:29:12 3: main - at wasm-micro-runtime/samples/debug-tools/wasm-apps/trap.c:24:5 + at /path/to/wasm-micro-runtime/samples/debug-tools/wasm-apps/trap.c:36:5 4: __main_void - at unknown:?:? + at ??:0 5: _start + at wasisdk://v25.0/build/sysroot/wasi-libc-wasm32-wasi/libc-bottom-half/crt/crt1-command.c:43:13 ``` -If WAMR is run in fast interpreter mode (`WAMR_BUILD_FAST_INTERP=1`), addresses in the stack trace cannot be tracked back to location info. -If WAMR <= `1.3.2` is used, the stack trace does not contain addresses. -In those two cases, run the script with `--no-addr`: the line info returned refers to the start of the function +Frame `0` shows **inline expansion** in action — `trap_helper` (the actual trap site) +and `c` (its caller) appear together under index `0` because `trap_helper` was inlined +into `c` by `__attribute__((always_inline))`. The `(inlined into c)` suffix on +`trap_helper` makes the relationship explicit: the runtime saw one WASM frame for `c`, +but addr2line.py reconstructs the full source-level call chain from DWARF +`DW_TAG_inlined_subroutine` entries. Each frame in the chain except the outermost +gets the `(inlined into )` annotation. + +### Inline expansion + +addr2line.py automatically expands inline call chains when the binary's DWARF contains +`DW_TAG_inlined_subroutine` entries. This happens when functions are inlined either by +optimization (e.g. `-O2`, `-Oz`) or by `__attribute__((always_inline))`. + +The included `trap.c` marks `trap_helper` as always_inline and places `__builtin_trap()` +inside it. The trap address therefore falls within the inlined region, producing the +multi-line frame `0` shown above. Each inlined frame's source location is reported +independently, so you can see exactly which inlined call chain led to the trap. + +Inline expansion requires no flags — it's automatic whenever the DWARF carries the +relevant inline metadata. + +For details on the tools `addr2line.py` orchestrates internally and why each +is needed, see [`test-tools/addr2line/README.md`](../../test-tools/addr2line/README.md). +The longer-form discussion of the interval-table overlay — applied to correct +the LLVM symbolizer's outermost function-name lookup on wasm — also lives there. + +### Execution modes — `--mode={interp,aot,fast-interp}` + +Different WAMR execution modes report call-stack offsets in different address spaces. +Pass the right `--mode` to addr2line.py so offsets are interpreted correctly: + +| Mode | When | Adjustment | Source resolution | +|------|------|-----------|-------------------| +| `interp` (default) | Classic interpreter (`WAMR_BUILD_FAST_INTERP=0`) | `offset - code_start - 1` (post-advance) | Full file:line, inline expansion | +| `aot` | AOT-compiled .aot files | `offset - code_start` (wamrc commits at instruction start) | Full file:line, inline expansion | +| `fast-interp` | Fast interpreter (`WAMR_BUILD_FAST_INTERP=1`) | (not mappable) | Function-name only | + +For AOT call stacks: + +```bash +$ python3 ../../../test-tools/addr2line/addr2line.py \ + --wasi-sdk /opt/wasi-sdk \ + --wabt /opt/wabt \ + --wasm-file wasm-apps/trap.wasm \ + --mode aot \ + call_stack_aot.txt +``` + +For fast-interp call stacks: + +```bash +$ python3 ../../../test-tools/addr2line/addr2line.py \ + --wasi-sdk /opt/wasi-sdk \ + --wabt /opt/wabt \ + --wasm-file wasm-apps/trap.wasm \ + --mode fast-interp \ + call_stack.txt +``` + +Fast-interp transforms the WASM bytecode in-memory at load time (replaces opcodes +with handler indices, expands LEBs to fixed widths). The runtime ip therefore +points into the *transformed* buffer — there's no way to map an offset back to a +source line. `--mode=fast-interp` falls back to function-name lookup (equivalent +to `--no-addr`). + +If WAMR <= `1.3.2` is used, the stack trace does not contain addresses at all. +In that case, run the script with `--no-addr`: the line info returned refers to +the start of the function. ```bash $ python3 ../../../test-tools/addr2line/addr2line.py \ diff --git a/samples/debug-tools/symbolicate.sh b/samples/debug-tools/symbolicate.sh old mode 100644 new mode 100755 index 709622f03d..e5a43e4c7e --- a/samples/debug-tools/symbolicate.sh +++ b/samples/debug-tools/symbolicate.sh @@ -1,23 +1,52 @@ #!/bin/bash -set -euox pipefail - -# Symbolicate .wasm -python3 ../../../test-tools/addr2line/addr2line.py \ - --wasi-sdk /opt/wasi-sdk \ - --wabt /opt/wabt \ - --wasm-file wasm-apps/trap.wasm \ - call_stack.txt - -# Symbolicate .wasm with `--no-addr` -python3 ../../../test-tools/addr2line/addr2line.py \ - --wasi-sdk /opt/wasi-sdk \ - --wabt /opt/wabt \ - --wasm-file wasm-apps/trap.wasm \ - call_stack.txt --no-addr - -# Symbolicate .aot -python3 ../../../test-tools/addr2line/addr2line.py \ - --wasi-sdk /opt/wasi-sdk \ - --wabt /opt/wabt \ - --wasm-file wasm-apps/trap.wasm \ - call_stack_aot.txt \ No newline at end of file +# Copyright (C) 2019 Intel Corporation. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception + +set -euo pipefail + +# Symbolicate the captured call stack files (call_stack.txt for .wasm, +# call_stack_aot.txt for .aot) located in this sample's build/ directory. +# Self-locating so invocation works from any cwd. + +SCRIPT_DIR=$(cd "$(dirname "$0")" && pwd) +BUILD_DIR="${SCRIPT_DIR}/build" +ADDR2LINE="${SCRIPT_DIR}/../../test-tools/addr2line/addr2line.py" +WASM_FILE="${BUILD_DIR}/wasm-apps/trap.wasm" +CALL_STACK="${BUILD_DIR}/call_stack.txt" +CALL_STACK_AOT="${BUILD_DIR}/call_stack_aot.txt" + +WASI_SDK_PATH="${WASI_SDK_PATH:-/opt/wasi-sdk}" +WABT_PATH="${WABT_PATH:-/opt/wabt}" + +banner() { + echo + echo "===== $* =====" +} + +# .wasm + classic-interp: addr2line.py defaults to --mode=interp. +banner "wasm / --mode=interp (default; classic interpreter, post-advance offsets)" +python3 "$ADDR2LINE" \ + --wasi-sdk "$WASI_SDK_PATH" \ + --wabt "$WABT_PATH" \ + --wasm-file "$WASM_FILE" \ + "$CALL_STACK" + +# Same call stack with --no-addr: function-level resolution only — use +# this for fast-interp call stacks (transformed bytecode has no source +# mapping) and for very old iwasm dumps that don't carry offsets. +banner "wasm / --no-addr (function-name lookup only; fast-interp & WAMR<=1.3.2 fallback)" +python3 "$ADDR2LINE" \ + --wasi-sdk "$WASI_SDK_PATH" \ + --wabt "$WABT_PATH" \ + --wasm-file "$WASM_FILE" \ + --no-addr \ + "$CALL_STACK" + +# .aot: --mode=aot — wamrc commits ip at instruction-start (not post-advance). +banner "aot / --mode=aot (instruction-start offsets)" +python3 "$ADDR2LINE" \ + --wasi-sdk "$WASI_SDK_PATH" \ + --wabt "$WABT_PATH" \ + --wasm-file "$WASM_FILE" \ + --mode aot \ + "$CALL_STACK_AOT" diff --git a/samples/debug-tools/verify.sh b/samples/debug-tools/verify.sh new file mode 100755 index 0000000000..59a148f70e --- /dev/null +++ b/samples/debug-tools/verify.sh @@ -0,0 +1,83 @@ +#!/bin/bash +# Copyright (C) 2019 Intel Corporation. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +# +# Verify symbolicated output for samples/debug-tools. +# +# Usage: ./verify.sh +# +# Captures call stacks for both .wasm and .aot, runs symbolicate.sh, and +# asserts that the inline expansion of trap_helper (always_inline) shows +# up under both --mode=interp (the default for the .wasm path) and +# --mode=aot (the third invocation in symbolicate.sh). + +set -euo pipefail + +SCRIPT_DIR=$(cd "$(dirname "$0")" && pwd) +BUILD_DIR="${SCRIPT_DIR}/build" + +if [ ! -x "${BUILD_DIR}/iwasm" ]; then + echo "iwasm not found at ${BUILD_DIR}/iwasm; build the sample first" >&2 + exit 1 +fi + +cd "${BUILD_DIR}" +./iwasm wasm-apps/trap.wasm 2>&1 | grep "^#" > call_stack.txt || true +./iwasm wasm-apps/trap.aot 2>&1 | grep "^#" > call_stack_aot.txt || true + +OUT=$(mktemp) +trap 'rm -f "$OUT"' EXIT +"${SCRIPT_DIR}/symbolicate.sh" 2>&1 | tee "$OUT" > /dev/null + +assert() { + local pattern="$1" + local where="$2" + if ! grep -q "$pattern" "$OUT"; then + echo "FAIL: pattern '$pattern' ($where) not found in symbolicate.sh output" >&2 + cat "$OUT" >&2 + exit 1 + fi +} + +# assert_re: like assert but with POSIX extended regex (anchored line match). +assert_re() { + local pattern="$1" + local where="$2" + if ! grep -Eq "$pattern" "$OUT"; then + echo "FAIL: regex '$pattern' ($where) did not match symbolicate.sh output" >&2 + cat "$OUT" >&2 + exit 1 + fi +} + +# trap.c's runtime call chain is _start -> ... -> main -> a -> b -> c +# (with c calling __builtin_trap inside trap_helper, an always_inline +# helper). Symbolicate.sh runs three invocations against the same call +# stack. We assert the precise frames each one produces. +# +# All invocations should symbolicate the user-code frames `c`, `b`, `a`, +# `main` (each on its own line, prefixed by the frame index). +assert_re '^[0-9]+: c$' "user frame: c" +assert_re '^[0-9]+: b$' "user frame: b" +assert_re '^[0-9]+: a$' "user frame: a" +assert_re '^[0-9]+: main$' "user frame: main" +assert "trap.c" "source file" + +# Inline expansion of trap_helper (always_inline) only surfaces in the +# address-based runs (default --mode=interp and --mode=aot). It MUST +# appear at least once in the captured output, with the +# "(inlined into c)" annotation that print_frames adds for inlined +# frames. +assert_re '^[[:space:]]*0:[[:space:]]+trap_helper[[:space:]]+\(inlined into c\)$' \ + "inline expansion: trap_helper (inlined into c)" + +# The AOT block is the third (last) invocation in symbolicate.sh. +# Confirm --mode=aot also produces inline expansion (the outermost +# frame at index 0 should still be trap_helper inlined into c). +if ! tail -25 "$OUT" | grep -Eq '^[[:space:]]*0:[[:space:]]+trap_helper[[:space:]]+\(inlined into c\)$'; then + echo "FAIL: AOT block (last 25 lines) missing 'trap_helper (inlined into c)' — --mode=aot may be broken" >&2 + cat "$OUT" >&2 + exit 1 +fi + +echo "PASS [debug-tools symbolication: precise frames + inline expansion verified for both wasm and aot]" diff --git a/samples/debug-tools/wasm-apps/CMakeLists.txt b/samples/debug-tools/wasm-apps/CMakeLists.txt index 9858b98dab..82ea845c1b 100644 --- a/samples/debug-tools/wasm-apps/CMakeLists.txt +++ b/samples/debug-tools/wasm-apps/CMakeLists.txt @@ -34,6 +34,9 @@ function (compile_sample SOURCE_FILE) COMMAND ${WAMRC_BIN} --size-level=0 --enable-dump-call-stack -o ${AOT_FILE} ${WASM_FILE} WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR} ) + # Force the .wasm producer to run before the AOT step under `make -j`; + # the DEPENDS line above only names the file, not the CMake target. + add_dependencies(${FILE_NAME}_aot ${FILE_NAME}) ## wasm + sourcemap if (DEFINED EMSCRIPTEN) diff --git a/samples/debug-tools/wasm-apps/trap.c b/samples/debug-tools/wasm-apps/trap.c index 364c430d19..27a5d0245b 100644 --- a/samples/debug-tools/wasm-apps/trap.c +++ b/samples/debug-tools/wasm-apps/trap.c @@ -1,7 +1,18 @@ +__attribute__((always_inline)) static inline int +trap_helper(int n) +{ + /* Forced-inline helper to demonstrate inline call stack expansion. + The trap is inside this helper so the runtime offset falls within + the inlined region — addr2line.py will then show both trap_helper + and c() in the call chain (innermost first). */ + __builtin_trap(); + return n + 100; +} + int c(int n) { - __builtin_trap(); + return trap_helper(n); } int