From f122d0289427dfe1eb26ef1a3c8aceed06bdc737 Mon Sep 17 00:00:00 2001 From: Greg Burd Date: Wed, 29 Jul 2026 06:16:41 -0400 Subject: [PATCH] test(upgrade): cover per-version upgrade transforms via synthetic old-format fixtures Extend test/db/run_upgrade.sh to exercise the on-disk-format upgrade code (src/db/db_upg.c, src/btree/bt_upgrade.c, src/hash/hash_upgrade.c, src/qam/qam_upgrade.c), which was almost entirely dead (0-14%) because this fork ships only one current-format fixture and lacks upstream's per-version Tcl upgrade fixture tree (test/tcl/upgrade/databases/). The driver now: - runs db_upgrade (DB->upgrade) + db_verify over the committed btree v9 fixture and freshly-created current-format dbs of every access method (btree/hash/queue/recno + btree-with-dups + the -s DUPSORT flag and salvage), covering the full magic/version dispatch and the version-current no-op branch for each AM; - manufactures old-format fixtures WITHOUT an old library, by rewriting a current db's metadata page into the byte layout of an older release (HASHHDR v5, HMETA30 v6, BTMETA2X v6, BTMETA30 v7, QMETA30 v1, QMETA31 v2 from dbinc/db_upgrade.h) with the version field set back, driving the real per-version transform functions. Coverage (line), before -> after: db/db_upg.c 13.9% -> 57.2% (functions 40% -> 100%) qam/qam_upgrade.c 0.0% -> 97.1% (functions 0% -> 100%) btree/bt_upgrade.c 0.0% -> 82.4% (functions 0% -> 100%) hash/hash_upgrade.c 0.0% -> 68.5% (functions 0% -> 66.7%) db/db_upg_opd.c 0.0% -> 0.0% (still cold; see below) Fixtures are generated at test time from current-format dbs, so no binary fixtures are committed. Two documented gaps (in test/coverage/README.md, engine NOT changed): - db_upg_opd.c (__db_31_offdup, 3.0->3.1 off-page-dup conversion) needs a genuine 3.0-era off-page-duplicate page chain (linked __P_DUPLICATE pages); a rewritten current db can't produce it (current dups are already a Recno tree), so it stays 0%. - __db_set_lastpgno off-by-one: the btree v6/v7 path stores __db_lastpgno()'s page COUNT into meta->last_pgno (a page NUMBER = count-1), so db_verify rejects last_pgno on a page-aligned file ("last_pgno is not correct: N != N-1", under HAVE_FTRUNCATE). The hash path avoids it (its v8->v9 pass runs through mpool, which recomputes last_pgno on close). The code is identical to upstream BDB 4.7/4.8. The driver upgrades btree v6/v7 (covering the transform) but skips db_verify on them, asserting the version bump instead. Reported to confirm against a genuine old fixture; not fixed. --- test/coverage/README.md | 40 ++++-- test/coverage/run_coverage.sh | 11 +- test/db/run_upgrade.sh | 241 +++++++++++++++++++++++++++++++--- 3 files changed, 264 insertions(+), 28 deletions(-) diff --git a/test/coverage/README.md b/test/coverage/README.md index 81176490e..6addc8323 100644 --- a/test/coverage/README.md +++ b/test/coverage/README.md @@ -55,14 +55,38 @@ timeout, so they cannot hang: harness needs an Oracle Tuxedo install (`atmi.h`, `tmboot`, …) which is not available; this driver needs none. Lift: `xa.c` **0%→~57%**, `xa_map.c` **0%→~78%**. -- **On-disk upgrade** — `test/db/run_upgrade.sh` runs the `db_upgrade` utility - over the one committed old-format fixture (`test/csharp/bdb4.7.db`, a btree - meta version-9 db) and verifies the result. Lift: `db_upg.c` **0%→~14%** - (the utility open + version-check + no-op-when-current path). `db_upg_opd.c` - stays 0%: the off-page-duplicate conversion needs a genuinely old - (pre-version-9) db with off-page dups, and the full Tcl `upgrade` group's - per-version fixture tree (`test/tcl/upgrade/databases/`) is **not present in - this fork**. See `DB-REPSITE-TODO.md` for the analogous repmgr gap. +- **On-disk upgrade** — `test/db/run_upgrade.sh` runs the `db_upgrade` + utility (which calls `DB->upgrade`) + `db_verify` over Berkeley DB files of + many on-disk versions. It uses the one committed old-format fixture + (`test/csharp/bdb4.7.db`, a btree meta version-9 db) plus freshly-created + current-format dbs of every access method, and -- since this fork lacks the + upstream Tcl `upgrade` per-version fixture tree + (`test/tcl/upgrade/databases/`) -- it manufactures old-format fixtures + *without an old library* by rewriting the metadata page of a current db into + the byte layout of an older release (BTMETA2X / BTMETA30 / HASHHDR / HMETA30 + / QMETA30 / QMETA31 from `dbinc/db_upgrade.h`) with the version field set + back. Lift: `qam_upgrade.c` **0%→~97%**, `hash_upgrade.c` **0%→~68%**, + `bt_upgrade.c` **0%→~82%**, `db_upg.c` **0%→~57%** (the whole magic/version + dispatch + real per-version transforms). + - `db_upg_opd.c` **still 0%**: the 3.0→3.1 off-page-duplicate conversion + (`__db_31_offdup`) needs a genuine 3.0-era off-page-duplicate page CHAIN + (linked `__P_DUPLICATE` pages). That cannot be produced by rewriting a + current db, because current off-page dups are already stored as a Recno + tree (P_LRECNO/P_IRECNO), not a flat page chain. A genuine pre-3.1 fixture + is required. + - **`__db_set_lastpgno` off-by-one (finding, not fixed):** the btree v6/v7 + upgrade path calls `__db_set_lastpgno()`, which stores `__db_lastpgno()`'s + result -- the page COUNT (`bytes/pagesize`) -- directly into + `meta->last_pgno`, the last page NUMBER (`count-1`). On a page-aligned file + that is off by +1, so `db_verify` reports `last_pgno is not correct: N != + N-1` (`db_vrfy.c`, under `HAVE_FTRUNCATE`). The hash path avoids it because + its v8→v9 pass runs through mpool, which recomputes `last_pgno` correctly + on close. The driver therefore upgrades the btree v6/v7 fixtures (to cover + the transform code) but skips `db_verify` on them, asserting the metadata + version bumped instead. `__db_set_lastpgno` is identical to upstream + Berkeley DB 4.7/4.8, so this is likely a long-standing latent defect; the + fixtures here are synthetic, so it is reported to confirm against a genuine + old fixture, not fixed. See `DB-REPSITE-TODO.md` for the analogous repmgr gap. Outputs land in `build_unix/` (gitignored): diff --git a/test/coverage/run_coverage.sh b/test/coverage/run_coverage.sh index 79ae81521..92757288f 100755 --- a/test/coverage/run_coverage.sh +++ b/test/coverage/run_coverage.sh @@ -208,10 +208,13 @@ fi # that drives the switch entry points + the internal 2PC / recovery path) # and runs it under a timeout. Lifts xa.c 0%->~57%, xa_map.c 0%->~78%. # db/db_upg.c -- the on-disk-format upgrade path. -# test/db/run_upgrade.sh runs the db_upgrade utility over the committed -# old-format fixture (test/csharp/bdb4.7.db). Lifts db_upg.c 0%->~14%. -# (db_upg_opd.c stays cold: needs an old off-page-duplicate fixture that is -# not present in this fork -- see test/coverage/README.md.) +# test/db/run_upgrade.sh runs the db_upgrade utility + DB->upgrade over the +# committed old-format fixture, freshly-created current-format dbs of every +# access method, AND synthetic old-format fixtures (metadata pages rewritten +# into BTMETA2X/BTMETA30/HASHHDR/HMETA30/QMETA30/QMETA31 layouts). Lifts +# db_upg.c 0%->~57%, qam_upgrade.c 0%->~97%, hash_upgrade.c 0%->~68%, +# bt_upgrade.c 0%->~82%. (db_upg_opd.c stays cold: needs a genuine 3.0-era +# off-page-duplicate page chain -- see test/coverage/README.md.) # Both self-clean their home dir and run under a hard timeout, so they cannot # hang. Set COV_XA_UPG=0 to skip. if [ "${COV_XA_UPG:-1}" = 1 ]; then diff --git a/test/db/run_upgrade.sh b/test/db/run_upgrade.sh index 05f9973a3..eaeaed342 100644 --- a/test/db/run_upgrade.sh +++ b/test/db/run_upgrade.sh @@ -4,20 +4,69 @@ # # run_upgrade.sh -- # Exercise the on-disk-format upgrade path (src/db/db_upg.c, -# db_upg_opd.c) by running the db_upgrade utility + DB->upgrade over an -# old-format Berkeley DB file. +# src/btree/bt_upgrade.c, src/hash/hash_upgrade.c, +# src/qam/qam_upgrade.c) by running the db_upgrade utility (which calls +# DB->upgrade) + db_verify over Berkeley DB files of many on-disk +# versions. # -# The full Tcl "upgrade" group (test/tcl/upgrade.tcl) needs a large -# per-version/per-method fixture tree under test/tcl/upgrade/databases/ -# that is NOT present in this fork. This script instead uses the one -# committed old-format fixture (test/csharp/bdb4.7.db, a btree meta -# version-9 database) to drive the utility's open/version-check path and -# DB->upgrade's no-op-when-current branch, then verifies the result. +# Berkeley DB never removes support for reading an older on-disk format, +# so the upgrade code still has per-version metadata/page transforms for +# btree v6/v7/v8, hash v4/v5/v6/v7/v8, and queue v1/v2 -- but the fork's +# tree ships only ONE current-format fixture, so almost all of that code +# was dead (0%). The genuine per-version fixture tree that upstream's +# Tcl "upgrade" group uses (test/tcl/upgrade/databases/) is not present. +# +# This driver builds old-format fixtures WITHOUT an old library: it +# creates a current-format database with the Tcl API and then rewrites +# *only its metadata page* into the byte layout of an older release +# (DBMETA30/DBMETA31/BTMETA2X/HASHHDR/QMETA30/QMETA31, from +# src/dbinc/db_upgrade.h) with the version field set back. Because the +# old->new metadata transforms operate on the metadata page in place (and +# for btree/queue never touch the data pages), a v9 file whose meta has +# been rewritten as a valid older meta is a legitimate input to +# __db_upgrade and drives the real transform functions. +# +# Coverage reached (lines): +# qam_upgrade.c ~97% (both __qam_31/32_qammeta, via queue v1 & v2) +# hash_upgrade.c ~68% (__ham_30/31/46_*, via hash v5 & v6) +# bt_upgrade.c ~82% (__bam_30/31_*, via btree v6 & v7) +# db_upg.c ~57% (the whole magic/version dispatch) +# +# NOT reachable here (documented in test/coverage/README.md): +# * db_upg_opd.c (the 3.0->3.1 off-page-duplicate conversion, +# __db_31_offdup) -- needs a genuine 3.0-era off-page-duplicate page +# CHAIN (linked __P_DUPLICATE pages), which cannot be produced by +# rewriting a current file (current dups are already a Recno tree). +# * db_verify AFTER a btree v6/v7 upgrade -- see the note on the +# __db_set_lastpgno off-by-one below. +# +# --- __db_set_lastpgno finding (documented, NOT fixed here) ------------------ +# The btree v7->v8 and v8->v9 steps call __db_set_lastpgno(), which stores +# __db_lastpgno()'s result -- the page COUNT (bytes/pagesize) -- directly +# into meta->last_pgno, which is the last page NUMBER (count-1). On any +# page-aligned file that is off by exactly +1, and db_verify then reports +# "last_pgno is not correct: N != N-1" (db_vrfy.c, under HAVE_FTRUNCATE). +# The hash path does NOT hit this because its v8->v9 pass runs through +# mpool, which recomputes last_pgno correctly on close. This driver +# therefore runs db_upgrade (rc must be 0) over the btree v6/v7 fixtures +# to cover the transform code, but does NOT db_verify them; it asserts the +# metadata version was bumped instead. The arithmetic is +# fixture-independent, but the fixtures here are synthetic, so this is +# reported as a finding to confirm against a genuine old fixture, not a +# fix. # # Usage (from build_unix): # sh ../test/db/run_upgrade.sh # -# Exits non-zero on failure. Runs under a timeout so it cannot hang. +# Optional environment: +# BUILD build_unix dir holding db_upgrade/db_verify (default: .) +# FIXTURE committed old-format fixture (default: ../test/csharp/bdb4.7.db) +# TCL_LIB libdb_tcl-*.so (auto-detected under $BUILD/.libs) +# TCLSH tclsh binary (default: tclsh8.6 if present, else tclsh) +# PYTHON python3 used to rewrite metadata pages (default: python3) +# +# Every db_upgrade/db_verify run is wrapped in a timeout so it cannot hang. +# Exits non-zero on failure. set -e @@ -25,22 +74,182 @@ BUILD=${BUILD:-.} FIXTURE=${FIXTURE:-../test/csharp/bdb4.7.db} WORK=${WORK:-UPGTEST} TIMEOUT=${TIMEOUT:-60} +PYTHON=${PYTHON:-python3} -[ -f "$FIXTURE" ] || { echo "FAIL: fixture $FIXTURE not found"; exit 1; } [ -x "$BUILD/db_upgrade" ] || { echo "FAIL: $BUILD/db_upgrade missing"; exit 1; } +[ -x "$BUILD/db_verify" ] || { echo "FAIL: $BUILD/db_verify missing"; exit 1; } ABS_UPGRADE=$(cd "$BUILD" && pwd)/db_upgrade ABS_VERIFY=$(cd "$BUILD" && pwd)/db_verify -ABS_FIXTURE=$(cd "$(dirname "$FIXTURE")" && pwd)/$(basename "$FIXTURE") rm -f "$WORK"/__db.* "$WORK"/log.* "$WORK"/*.db 2>/dev/null || true mkdir -p "$WORK" -cp "$ABS_FIXTURE" "$WORK/t.db" -echo "db_upgrade on $(basename "$FIXTURE")" -( cd "$WORK" && timeout "$TIMEOUT" "$ABS_UPGRADE" -h . t.db ) -echo "db_verify on upgraded file" -( cd "$WORK" && timeout "$TIMEOUT" "$ABS_VERIFY" t.db ) +# upgrade_verify FILE [extra db_upgrade flags...] -- run db_upgrade then +# db_verify (both under a timeout). Used for paths that must verify clean. +upgrade_verify() { + f=$1; shift + echo "db_upgrade $* + verify: $f" + ( cd "$WORK" && timeout "$TIMEOUT" "$ABS_UPGRADE" -h . "$@" "$f" ) + ( cd "$WORK" && timeout "$TIMEOUT" "$ABS_VERIFY" "$f" >/dev/null ) + echo " CLEAN $f" +} + +# upgrade_only FILE EXPECTVER -- run db_upgrade (must succeed) and assert the +# metadata version was bumped to EXPECTVER. Used for btree v6/v7 where the +# documented __db_set_lastpgno off-by-one makes db_verify (correctly) reject +# last_pgno; we still want the transform code covered. +upgrade_only() { + f=$1; exp=$2 + echo "db_upgrade (no verify): $f" + ( cd "$WORK" && timeout "$TIMEOUT" "$ABS_UPGRADE" -h . "$f" ) + got=$( cd "$WORK" && "$PYTHON" -c "import struct,sys;print(struct.unpack_from('$got; verify skipped: __db_set_lastpgno off-by-one)" +} + +# ---- 1. committed old-format btree v9 fixture (no-op dispatch) -------------- +if [ -f "$FIXTURE" ]; then + cp "$(cd "$(dirname "$FIXTURE")" && pwd)/$(basename "$FIXTURE")" "$WORK/fixture.db" + upgrade_verify fixture.db +else + echo "note: fixture $FIXTURE not found, skipping" +fi + +# ---- 2. current-format + synthetic old-format DBs via the Tcl API ----------- +if [ -z "${TCL_LIB:-}" ]; then + TCL_LIB=$(ls "$BUILD"/.libs/libdb_tcl-*.so 2>/dev/null | head -1) +fi +TCLBIN=${TCLSH:-} +if [ -z "$TCLBIN" ]; then + command -v tclsh8.6 >/dev/null 2>&1 && TCLBIN=tclsh8.6 + [ -z "$TCLBIN" ] && command -v tclsh >/dev/null 2>&1 && TCLBIN=tclsh +fi + +if [ -z "$TCL_LIB" ] || [ ! -f "$TCL_LIB" ] || [ -z "$TCLBIN" ]; then + echo "note: libdb_tcl or tclsh not found, skipping current/old DB pass" + echo "run_upgrade.sh: PASS" + exit 0 +fi +ABS_TCL_LIB=$(cd "$(dirname "$TCL_LIB")" && pwd)/$(basename "$TCL_LIB") + +echo "creating current-format base DBs via $TCLBIN" +( cd "$WORK" && "$TCLBIN" </dev/null ) +echo " CLEAN cur_btdup.db (salvage)" + +# 2b. rewrite metadata pages into old on-disk layouts (see header). +( cd "$WORK" && "$PYTHON" - <<'PY' +import struct +def rd(p): return bytearray(open(p,'rb').read()) +def wr(p,b): open(p,'wb').write(b) + +# ---- hash: HASHHDR (v5) and HMETA30 (v6) ---- +b=rd('hbase.db'); psz=struct.unpack_from('