From 1a88ff32452368333adb98701af0b3b274fe885b Mon Sep 17 00:00:00 2001 From: Greg Burd Date: Wed, 29 Jul 2026 07:06:13 -0400 Subject: [PATCH] test(coverage): wire deadlock-detector + DB_REGISTER drivers into run_coverage.sh Add a COV_DEAD_REG block (on by default) that runs the two multi-process suites reaching the deadlock detector and the process-registry crash/ recovery path, which the single-tclsh COV_TESTS loop cannot: - dead001..dead006 spawn ddscript.tcl workers via wrap.tcl to build a ring/clump lock cycle; __lock_detect (src/lock/lock_deadlock.c) then picks a victim. dead002/003 run detection in-process in each worker. - env007 + env012 use envscript.tcl to open the env with DB_REGISTER, "crash" a process, and let a survivor detect the dead slot (__envreg_register/__envreg_isalive, src/env/env_register.c) and run recovery/failchk. Like the COV_REP block these run driver-per-test with a per-test timeout and orphan-worker cleanup (they reset TESTDIR and a hung worker must not wedge the run). Proc counts are trimmed to {2 4} ({4} for dead005) so each finishes well inside the 300s timeout; the full {2 4 10} matrix adds only minutes, no new lines. Measured on r-class box (gcc --coverage, lcov from .libs): lock/lock_deadlock.c 0.7% -> 65.9% line (43.2% branch) env/env_register.c 0.0% -> 54.7% line (43.1% branch) env/env_failchk.c 17.4% -> 65.2% line (bonus, DB_REGISTER failchk path) All eight tests pass reliably under timeout. README documents the block and updates the least-covered table. --- test/coverage/README.md | 26 +++++++++++++++++-- test/coverage/run_coverage.sh | 49 +++++++++++++++++++++++++++++++++++ 2 files changed, 73 insertions(+), 2 deletions(-) diff --git a/test/coverage/README.md b/test/coverage/README.md index dfb32127b..dc10325b4 100644 --- a/test/coverage/README.md +++ b/test/coverage/README.md @@ -35,6 +35,7 @@ Knobs (all optional env vars): | `COV_TIMEOUT`| `2400` | seconds ceiling for the test run | | `COV_REP` | `0` | set `1` to also run the replication (rep/repmgr) tests | | `COV_XA_UPG` | `1` | run the XA + on-disk-upgrade drivers (fast, non-hanging) | +| `COV_DEAD_REG` | `1` | run the deadlock-detector + DB_REGISTER multi-process tests | Set `COV_REP=1` to add the replication suite (biggest cold surface: rep/ + repmgr/ ~= 12.4k lines at 0.8%). It moves them to ~56% line / ~42% branch. Those @@ -131,6 +132,26 @@ Tcl tests now drive the whole surface (both in the default `COV_TESTS`): binding exposes only `-clear`, not `-all`); `repmgr_stat.c` needs a live repmgr transport (`repmgr009`+, `COV_REP=1`) for its send-path counters. +`COV_DEAD_REG` (on by default) runs the two multi-process suites that reach the +deadlock detector and the process-registry crash/recovery path. Like `COV_REP`, +they run driver-per-test with a per-test timeout and orphan-worker cleanup +(they reset `TESTDIR` and each spawns child `tclsh` via `wrap.tcl`, so a hung +worker must not wedge the whole run): + +- **Deadlock detector** — the `dead` group (`dead001`–`dead006`) spawns + `ddscript.tcl` workers that grab locks in a ring/clump cycle; the detector + (`src/lock/lock_deadlock.c`, `__lock_detect`) then picks a victim. `dead002`/ + `dead003` set `-lock_detect` so detection runs **in-process** in each worker + (which flushes its own `.gcda`); `dead001`/`dead004`/`dead005` use the + standalone `db_deadlock` utility. Proc counts are trimmed to `{2 4}` (`{4}` + for `dead005`) — the full `{2 4 10}` matrix adds only minutes, no new lines. + Lift: `lock_deadlock.c` **0.7%→~66%**. +- **DB_REGISTER** — `env012` (with `env007`) uses `envscript.tcl` to open the + env with `DB_REGISTER`, "crash" a process (kill without close), then reopen + with `-recover`/`-failchk` so the survivor detects the dead slot + (`src/env/env_register.c`, `__envreg_register`/`__envreg_isalive`) and runs + recovery. Lift: `env_register.c` **0%→~55%**. + Outputs land in `build_unix/` (gitignored): - `coverage-summary.txt` — the line/branch/function totals @@ -265,11 +286,12 @@ why they lag the single-process access-method tests: | 0.0 | 0.0 | 701 | 556 | `repmgr/repmgr_msg.c` | replication manager | | 0.0 | 0.0 | 556 | 656 | `rep/rep_elect.c` | replication elections | | 0.0 | 0.0 | 541 | 460 | `rep/rep_automsg.c` | replication | -| 0.0 | 0.0 | 457 | 674 | `lock/lock_deadlock.c` | deadlock detection | | 0.0 | 0.0 | 431 | 470 | `sequence/sequence.c` | sequences | -| 0.0 | 0.0 | 394 | 398 | `xa/xa.c` | XA transactions | | 5.4 | 2.0 | 445 | 306 | `qam/qam_files.c` | queue extent files | | 30.1 | 17.4 | 1397 | 1702 | `btree/bt_compact.c` | btree compaction (was 0%) | +| 54.7 | 43.1 | 223 | 174 | `env/env_register.c` | DB_REGISTER / failchk (was 0%, COV_DEAD_REG) | +| 57.1 | ~ | 394 | 398 | `xa/xa.c` | XA transactions (was 0%, COV_XA_UPG) | +| 65.9 | 43.3 | 457 | 674 | `lock/lock_deadlock.c` | deadlock detection (was 0.7%, COV_DEAD_REG) | Next-highest-leverage additions (future work): a **replication smoke test** (needs the multi-process rep harness — `rep0NN` / `repmgr0NN` exist but each diff --git a/test/coverage/run_coverage.sh b/test/coverage/run_coverage.sh index edfb49479..32ddceb3b 100755 --- a/test/coverage/run_coverage.sh +++ b/test/coverage/run_coverage.sh @@ -242,6 +242,55 @@ if [ "${COV_XA_UPG:-1}" = 1 ]; then echo " .gcda files after xa/upg: $(find . -name '*.gcda' | wc -l)" fi +# --- optional deadlock-detector + DB_REGISTER drivers (COV_DEAD_REG=1) ------- +# The deadlock DETECTOR (lock/lock_deadlock.c, __lock_detect) and the +# process-registration + failchk-on-crash path (env/env_register.c, +# __envreg_register/isalive) are both multi-process: the `dead` group and +# env012 spawn worker tclsh via wrap.tcl (ddscript.tcl / envscript.tcl) that +# contend for locks in a cycle (detector picks a victim) or open the env with +# DB_REGISTER, crash, and let a survivor recover. They cannot run in the main +# single-tclsh COV_TESTS loop (they reset TESTDIR and a hung worker would wedge +# the whole run), so -- like COV_REP -- they run driver-per-test with a +# per-test timeout and orphan-worker cleanup. Lifts lock_deadlock.c 0.7%->~66% +# and env_register.c 0%->~55%. proc counts are trimmed (default {2 4}) so each +# test finishes well inside the timeout; the full {2 4 10} matrix adds no new +# coverage, only minutes. Set COV_DEAD_REG=0 to skip. +if [ "${COV_DEAD_REG:-1}" = 1 ]; then + echo "== run deadlock + DB_REGISTER drivers (COV_DEAD_REG=1) ==" + : "${COV_DEAD_REG_TIMEOUT:=300}" + # Each entry is "name:tcl-call"; trimmed proc counts keep each run bounded. + # A bash array so the spaces inside a call ({2 4}) survive word-splitting. + cov_dead_reg_tests=( + "dead001:dead001 {2 4}" + "dead002:dead002 {2 4}" + "dead003:dead003 {2 4}" + "dead004:dead004" + "dead005:dead005 {4}" + "dead006:dead006 {2 4}" + "env007:env007" + "env012:env012" + ) + dregtcl="$bld/.cov-deadreg-one.tcl" + for spec in "${cov_dead_reg_tests[@]}"; do + name="${spec%%:*}"; call="${spec#*:}" + printf 'source ../test/tcl/test.tcl\nif {[catch {%s} r]} { puts "FAIL %s: $r"; exit 3 }\nputs "PASS %s"\n' \ + "$call" "$name" "$name" > "$dregtcl" + timeout "$COV_DEAD_REG_TIMEOUT" "$TCLBIN" "$dregtcl" >/tmp/cov-dreg-$name.log 2>&1 + rc=$? + # kill orphan workers a hung/killed test may have left behind + pkill -f 'wrap.tcl' 2>/dev/null || true + pkill -f 'ddscript' 2>/dev/null || true + pkill -f 'envscript' 2>/dev/null || true + pkill -f 'db_deadlock' 2>/dev/null || true + if [ $rc -eq 124 ]; then echo "HANG $name" + elif [ $rc -eq 0 ] && grep -q "^PASS $name" /tmp/cov-dreg-$name.log; then echo "PASS $name" + else echo "FAIL $name (rc=$rc)"; fi + find TESTDIR -mindepth 1 -delete 2>/dev/null || true + done + rm -f "$dregtcl" + echo " .gcda files after dead/register: $(find . -name '*.gcda' | wc -l)" +fi + # --- aggregate --------------------------------------------------------------- echo "== capture (lcov) ==" # NOTE: capture from .libs (not .) -- libtool double-compiles, and only the