Skip to content

test(upgrade): cover per-version on-disk upgrade transforms via synthetic old-format fixtures #73

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
gburd wants to merge 1 commit into from
Closed

test(upgrade): cover per-version on-disk upgrade transforms via synthetic old-format fixtures #73

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 32 additions & 8 deletions test/coverage/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,14 +55,38 @@ timeout, so they cannot hang:
harness needs an Oracle Tuxedo install (`atmi.h`, `tmboot`, …) which is not
available; this driver needs none. Lift: `xa.c` **0%→~57%**, `xa_map.c`
**0%→~78%**.
- **On-disk upgrade** — `test/db/run_upgrade.sh` runs the `db_upgrade` utility
over the one committed old-format fixture (`test/csharp/bdb4.7.db`, a btree
meta version-9 db) and verifies the result. Lift: `db_upg.c` **0%→~14%**
(the utility open + version-check + no-op-when-current path). `db_upg_opd.c`
stays 0%: the off-page-duplicate conversion needs a genuinely old
(pre-version-9) db with off-page dups, and the full Tcl `upgrade` group's
per-version fixture tree (`test/tcl/upgrade/databases/`) is **not present in
this fork**. See `DB-REPSITE-TODO.md` for the analogous repmgr gap.
- **On-disk upgrade** — `test/db/run_upgrade.sh` runs the `db_upgrade`
utility (which calls `DB->upgrade`) + `db_verify` over Berkeley DB files of
many on-disk versions. It uses the one committed old-format fixture
(`test/csharp/bdb4.7.db`, a btree meta version-9 db) plus freshly-created
current-format dbs of every access method, and -- since this fork lacks the
upstream Tcl `upgrade` per-version fixture tree
(`test/tcl/upgrade/databases/`) -- it manufactures old-format fixtures
*without an old library* by rewriting the metadata page of a current db into
the byte layout of an older release (BTMETA2X / BTMETA30 / HASHHDR / HMETA30
/ QMETA30 / QMETA31 from `dbinc/db_upgrade.h`) with the version field set
back. Lift: `qam_upgrade.c` **0%→~97%**, `hash_upgrade.c` **0%→~68%**,
`bt_upgrade.c` **0%→~82%**, `db_upg.c` **0%→~57%** (the whole magic/version
dispatch + real per-version transforms).
- `db_upg_opd.c` **still 0%**: the 3.0→3.1 off-page-duplicate conversion
(`__db_31_offdup`) needs a genuine 3.0-era off-page-duplicate page CHAIN
(linked `__P_DUPLICATE` pages). That cannot be produced by rewriting a
current db, because current off-page dups are already stored as a Recno
tree (P_LRECNO/P_IRECNO), not a flat page chain. A genuine pre-3.1 fixture
is required.
- **`__db_set_lastpgno` off-by-one (finding, not fixed):** the btree v6/v7
upgrade path calls `__db_set_lastpgno()`, which stores `__db_lastpgno()`'s
result -- the page COUNT (`bytes/pagesize`) -- directly into
`meta->last_pgno`, the last page NUMBER (`count-1`). On a page-aligned file
that is off by +1, so `db_verify` reports `last_pgno is not correct: N !=
N-1` (`db_vrfy.c`, under `HAVE_FTRUNCATE`). The hash path avoids it because
its v8→v9 pass runs through mpool, which recomputes `last_pgno` correctly
on close. The driver therefore upgrades the btree v6/v7 fixtures (to cover
the transform code) but skips `db_verify` on them, asserting the metadata
version bumped instead. `__db_set_lastpgno` is identical to upstream
Berkeley DB 4.7/4.8, so this is likely a long-standing latent defect; the
fixtures here are synthetic, so it is reported to confirm against a genuine
old fixture, not fixed. See `DB-REPSITE-TODO.md` for the analogous repmgr gap.

Outputs land in `build_unix/` (gitignored):

Expand Down
11 changes: 7 additions & 4 deletions test/coverage/run_coverage.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -208,10 +208,13 @@ fi
# that drives the switch entry points + the internal 2PC / recovery path)
# and runs it under a timeout. Lifts xa.c 0%->~57%, xa_map.c 0%->~78%.
# db/db_upg.c -- the on-disk-format upgrade path.
# test/db/run_upgrade.sh runs the db_upgrade utility over the committed
# old-format fixture (test/csharp/bdb4.7.db). Lifts db_upg.c 0%->~14%.
# (db_upg_opd.c stays cold: needs an old off-page-duplicate fixture that is
# not present in this fork -- see test/coverage/README.md.)
# test/db/run_upgrade.sh runs the db_upgrade utility + DB->upgrade over the
# committed old-format fixture, freshly-created current-format dbs of every
# access method, AND synthetic old-format fixtures (metadata pages rewritten
# into BTMETA2X/BTMETA30/HASHHDR/HMETA30/QMETA30/QMETA31 layouts). Lifts
# db_upg.c 0%->~57%, qam_upgrade.c 0%->~97%, hash_upgrade.c 0%->~68%,
# bt_upgrade.c 0%->~82%. (db_upg_opd.c stays cold: needs a genuine 3.0-era
# off-page-duplicate page chain -- see test/coverage/README.md.)
# Both self-clean their home dir and run under a hard timeout, so they cannot
# hang. Set COV_XA_UPG=0 to skip.
if [ "${COV_XA_UPG:-1}" = 1 ]; then
Expand Down
241 changes: 225 additions & 16 deletions test/db/run_upgrade.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,43 +4,252 @@
#
# run_upgrade.sh --
# Exercise the on-disk-format upgrade path (src/db/db_upg.c,
# db_upg_opd.c) by running the db_upgrade utility + DB->upgrade over an
# old-format Berkeley DB file.
# src/btree/bt_upgrade.c, src/hash/hash_upgrade.c,
# src/qam/qam_upgrade.c) by running the db_upgrade utility (which calls
# DB->upgrade) + db_verify over Berkeley DB files of many on-disk
# versions.
#
# The full Tcl "upgrade" group (test/tcl/upgrade.tcl) needs a large
# per-version/per-method fixture tree under test/tcl/upgrade/databases/
# that is NOT present in this fork. This script instead uses the one
# committed old-format fixture (test/csharp/bdb4.7.db, a btree meta
# version-9 database) to drive the utility's open/version-check path and
# DB->upgrade's no-op-when-current branch, then verifies the result.
# Berkeley DB never removes support for reading an older on-disk format,
# so the upgrade code still has per-version metadata/page transforms for
# btree v6/v7/v8, hash v4/v5/v6/v7/v8, and queue v1/v2 -- but the fork's
# tree ships only ONE current-format fixture, so almost all of that code
# was dead (0%). The genuine per-version fixture tree that upstream's
# Tcl "upgrade" group uses (test/tcl/upgrade/databases/) is not present.
#
# This driver builds old-format fixtures WITHOUT an old library: it
# creates a current-format database with the Tcl API and then rewrites
# *only its metadata page* into the byte layout of an older release
# (DBMETA30/DBMETA31/BTMETA2X/HASHHDR/QMETA30/QMETA31, from
# src/dbinc/db_upgrade.h) with the version field set back. Because the
# old->new metadata transforms operate on the metadata page in place (and
# for btree/queue never touch the data pages), a v9 file whose meta has
# been rewritten as a valid older meta is a legitimate input to
# __db_upgrade and drives the real transform functions.
#
# Coverage reached (lines):
# qam_upgrade.c ~97% (both __qam_31/32_qammeta, via queue v1 & v2)
# hash_upgrade.c ~68% (__ham_30/31/46_*, via hash v5 & v6)
# bt_upgrade.c ~82% (__bam_30/31_*, via btree v6 & v7)
# db_upg.c ~57% (the whole magic/version dispatch)
#
# NOT reachable here (documented in test/coverage/README.md):
# * db_upg_opd.c (the 3.0->3.1 off-page-duplicate conversion,
# __db_31_offdup) -- needs a genuine 3.0-era off-page-duplicate page
# CHAIN (linked __P_DUPLICATE pages), which cannot be produced by
# rewriting a current file (current dups are already a Recno tree).
# * db_verify AFTER a btree v6/v7 upgrade -- see the note on the
# __db_set_lastpgno off-by-one below.
#
# --- __db_set_lastpgno finding (documented, NOT fixed here) ------------------
# The btree v7->v8 and v8->v9 steps call __db_set_lastpgno(), which stores
# __db_lastpgno()'s result -- the page COUNT (bytes/pagesize) -- directly
# into meta->last_pgno, which is the last page NUMBER (count-1). On any
# page-aligned file that is off by exactly +1, and db_verify then reports
# "last_pgno is not correct: N != N-1" (db_vrfy.c, under HAVE_FTRUNCATE).
# The hash path does NOT hit this because its v8->v9 pass runs through
# mpool, which recomputes last_pgno correctly on close. This driver
# therefore runs db_upgrade (rc must be 0) over the btree v6/v7 fixtures
# to cover the transform code, but does NOT db_verify them; it asserts the
# metadata version was bumped instead. The arithmetic is
# fixture-independent, but the fixtures here are synthetic, so this is
# reported as a finding to confirm against a genuine old fixture, not a
# fix.
#
# Usage (from build_unix):
# sh ../test/db/run_upgrade.sh
#
# Exits non-zero on failure. Runs under a timeout so it cannot hang.
# Optional environment:
# BUILD build_unix dir holding db_upgrade/db_verify (default: .)
# FIXTURE committed old-format fixture (default: ../test/csharp/bdb4.7.db)
# TCL_LIB libdb_tcl-*.so (auto-detected under $BUILD/.libs)
# TCLSH tclsh binary (default: tclsh8.6 if present, else tclsh)
# PYTHON python3 used to rewrite metadata pages (default: python3)
#
# Every db_upgrade/db_verify run is wrapped in a timeout so it cannot hang.
# Exits non-zero on failure.

set -e

BUILD=${BUILD:-.}
FIXTURE=${FIXTURE:-../test/csharp/bdb4.7.db}
WORK=${WORK:-UPGTEST}
TIMEOUT=${TIMEOUT:-60}
PYTHON=${PYTHON:-python3}

[ -f "$FIXTURE" ] || { echo "FAIL: fixture $FIXTURE not found"; exit 1; }
[ -x "$BUILD/db_upgrade" ] || { echo "FAIL: $BUILD/db_upgrade missing"; exit 1; }
[ -x "$BUILD/db_verify" ] || { echo "FAIL: $BUILD/db_verify missing"; exit 1; }

ABS_UPGRADE=$(cd "$BUILD" && pwd)/db_upgrade
ABS_VERIFY=$(cd "$BUILD" && pwd)/db_verify
ABS_FIXTURE=$(cd "$(dirname "$FIXTURE")" && pwd)/$(basename "$FIXTURE")

rm -f "$WORK"/__db.* "$WORK"/log.* "$WORK"/*.db 2>/dev/null || true
mkdir -p "$WORK"
cp "$ABS_FIXTURE" "$WORK/t.db"

echo "db_upgrade on $(basename "$FIXTURE")"
( cd "$WORK" && timeout "$TIMEOUT" "$ABS_UPGRADE" -h . t.db )
echo "db_verify on upgraded file"
( cd "$WORK" && timeout "$TIMEOUT" "$ABS_VERIFY" t.db )
# upgrade_verify FILE [extra db_upgrade flags...] -- run db_upgrade then
# db_verify (both under a timeout). Used for paths that must verify clean.
upgrade_verify() {
f=$1; shift
echo "db_upgrade $* + verify: $f"
( cd "$WORK" && timeout "$TIMEOUT" "$ABS_UPGRADE" -h . "$@" "$f" )
( cd "$WORK" && timeout "$TIMEOUT" "$ABS_VERIFY" "$f" >/dev/null )
echo " CLEAN $f"
}

# upgrade_only FILE EXPECTVER -- run db_upgrade (must succeed) and assert the
# metadata version was bumped to EXPECTVER. Used for btree v6/v7 where the
# documented __db_set_lastpgno off-by-one makes db_verify (correctly) reject
# last_pgno; we still want the transform code covered.
upgrade_only() {
f=$1; exp=$2
echo "db_upgrade (no verify): $f"
( cd "$WORK" && timeout "$TIMEOUT" "$ABS_UPGRADE" -h . "$f" )
got=$( cd "$WORK" && "$PYTHON" -c "import struct,sys;print(struct.unpack_from('<I',open(sys.argv[1],'rb').read(20),16)[0])" "$f" )
[ "$got" = "$exp" ] || { echo "FAIL: $f upgraded to version $got, expected $exp"; exit 1; }
echo " UPG-OK $f (version->$got; verify skipped: __db_set_lastpgno off-by-one)"
}

# ---- 1. committed old-format btree v9 fixture (no-op dispatch) --------------
if [ -f "$FIXTURE" ]; then
cp "$(cd "$(dirname "$FIXTURE")" && pwd)/$(basename "$FIXTURE")" "$WORK/fixture.db"
upgrade_verify fixture.db
else
echo "note: fixture $FIXTURE not found, skipping"
fi

# ---- 2. current-format + synthetic old-format DBs via the Tcl API -----------
if [ -z "${TCL_LIB:-}" ]; then
TCL_LIB=$(ls "$BUILD"/.libs/libdb_tcl-*.so 2>/dev/null | head -1)
fi
TCLBIN=${TCLSH:-}
if [ -z "$TCLBIN" ]; then
command -v tclsh8.6 >/dev/null 2>&1 && TCLBIN=tclsh8.6
[ -z "$TCLBIN" ] && command -v tclsh >/dev/null 2>&1 && TCLBIN=tclsh
fi

if [ -z "$TCL_LIB" ] || [ ! -f "$TCL_LIB" ] || [ -z "$TCLBIN" ]; then
echo "note: libdb_tcl or tclsh not found, skipping current/old DB pass"
echo "run_upgrade.sh: PASS"
exit 0
fi
ABS_TCL_LIB=$(cd "$(dirname "$TCL_LIB")" && pwd)/$(basename "$TCL_LIB")

echo "creating current-format base DBs via $TCLBIN"
( cd "$WORK" && "$TCLBIN" <<TCL
load $ABS_TCL_LIB
foreach {m fn} {btree cur_btree.db hash cur_hash.db recno cur_recno.db} {
set db [berkdb open -create -\$m -mode 0644 \$fn]
for {set i 1} {\$i <= 20} {incr i} {
if {\$m eq "recno"} { \$db put \$i "data_\$i" } \
else { \$db put "key_\$i" "data_\$i" }
}
\$db close
}
set db [berkdb open -create -queue -len 16 -mode 0644 cur_queue.db]
for {set i 1} {\$i <= 40} {incr i} { \$db put \$i [format "%-16s" rec_\$i] }
\$db close
# btree with sorted off-page duplicates
set db [berkdb open -create -btree -dup -dupsort -mode 0644 cur_btdup.db]
for {set i 1} {\$i <= 200} {incr i} { \$db put "dupkey" [format "v%05d" \$i] }
\$db close
# separate bases we will rewrite into old metadata layouts
set db [berkdb open -create -hash -mode 0644 hbase.db]
for {set i 1} {\$i <= 60} {incr i} { \$db put [format "hkey%04d" \$i] "hashval_\$i" }
\$db close
set db [berkdb open -create -btree -mode 0644 bbase.db]
for {set i 1} {\$i <= 30} {incr i} { \$db put [format "k%03d" \$i] "data_value_\$i" }
\$db close
set db [berkdb open -create -queue -len 16 -mode 0644 qbase.db]
for {set i 1} {\$i <= 40} {incr i} { \$db put \$i [format "%-16s" rec_\$i] }
\$db close
puts "created base DBs"
TCL
)

# 2a. current-format: db_upgrade is a no-op; exercises the whole dispatch +
# version-current "break" branch for btree/hash/queue/recno.
for f in cur_btree.db cur_hash.db cur_queue.db cur_recno.db; do
upgrade_verify "$f"
done
# btree-with-dups: also drive the -s (DB_DUPSORT) flag path + salvage.
upgrade_verify cur_btdup.db -s
echo "salvage-verify cur_btdup.db"
( cd "$WORK" && timeout "$TIMEOUT" "$ABS_VERIFY" -o cur_btdup.db >/dev/null )
echo " CLEAN cur_btdup.db (salvage)"

# 2b. rewrite metadata pages into old on-disk layouts (see header).
( cd "$WORK" && "$PYTHON" - <<'PY'
import struct
def rd(p): return bytearray(open(p,'rb').read())
def wr(p,b): open(p,'wb').write(b)

# ---- hash: HASHHDR (v5) and HMETA30 (v6) ----
b=rd('hbase.db'); psz=struct.unpack_from('<I',b,20)[0]
maxb,high,low,ff,nel,hck=struct.unpack_from('<6I',b,72) # HMETA33 body
spares=b[96:96+128]
free=struct.unpack_from('<I',b,28)[0]; flags=struct.unpack_from('<I',b,48)[0]
uid=bytes(b[52:72])
# v5: HASHHDR = lsn,pgno,magic(0-15) ver(16) psz(20) ovfl_point(24) last_freed(28)
# max_bucket..h_charkey(32-55) flags(56) spares[32](60-187) uid(188-207)
m=bytearray(psz); m[0:16]=b[0:16]
struct.pack_into('<I',m,16,5); struct.pack_into('<I',m,20,psz)
struct.pack_into('<I',m,24,0); struct.pack_into('<I',m,28,free)
struct.pack_into('<6I',m,32,maxb,high,low,ff,nel,hck)
struct.pack_into('<I',m,56,flags); m[60:188]=spares
wr('h_v5.db',bytes(m)+bytes(b[psz:]))
# v6: HMETA30 = DBMETA30 header(56) + max_bucket..h_charkey(56-79) + spares[32](80-207)
m=bytearray(psz); m[0:16]=b[0:16]
struct.pack_into('<I',m,16,6); struct.pack_into('<I',m,20,psz); m[25]=8 # P_HASHMETA
struct.pack_into('<I',m,28,free); struct.pack_into('<I',m,32,flags); m[36:56]=uid
struct.pack_into('<6I',m,56,maxb,high,low,ff,nel,hck); m[80:208]=spares
wr('h_v6.db',bytes(m)+bytes(b[psz:]))

# ---- btree: BTMETA2X (v6) and BTMETA30 (v7) ----
b=rd('bbase.db'); psz=struct.unpack_from('<I',b,20)[0]
minkey=struct.unpack_from('<I',b,76)[0]; re_len=struct.unpack_from('<I',b,80)[0]
re_pad=struct.unpack_from('<I',b,84)[0]; root=struct.unpack_from('<I',b,88)[0]
free=struct.unpack_from('<I',b,28)[0]; flags=struct.unpack_from('<I',b,48)[0]
uid=bytes(b[52:72])
# v6: BTMETA2X = header(0-23) maxkey(24) minkey(28) free(32) flags(36) re_len(40) re_pad(44) uid(48-67)
m=bytearray(psz); m[0:16]=b[0:16]
struct.pack_into('<I',m,16,6); struct.pack_into('<I',m,20,psz)
struct.pack_into('<6I',m,24,0,minkey,free,flags,re_len,re_pad); m[48:68]=uid
wr('b_v6.db',bytes(m)+bytes(b[psz:]))
# v7: BTMETA30 = DBMETA30 header(56) + maxkey(56) minkey(60) re_len(64) re_pad(68) root(72)
m=bytearray(psz); m[0:16]=b[0:16]
struct.pack_into('<I',m,16,7); struct.pack_into('<I',m,20,psz); m[25]=9 # P_BTREEMETA
struct.pack_into('<I',m,28,free); struct.pack_into('<I',m,32,flags); m[36:56]=uid
struct.pack_into('<5I',m,56,0,minkey,re_len,re_pad,root)
wr('b_v7.db',bytes(m)+bytes(b[psz:]))

# ---- queue: QMETA30 (v1) and QMETA31 (v2) ----
# original body is first_recno,cur_recno,re_len,re_pad,rec_page,page_ext at 72.
b=rd('qbase.db'); psz=struct.unpack_from('<I',b,20)[0]
_,cur,rl,rp,rpg,_=struct.unpack_from('<6I',b,72)
# __qam_32_qammeta does cur_recno++, so seed cur = cur-1 to land back on `cur`.
seed=cur-1
uid=bytes(b[52:72])
# v2: QMETA31 = header(72) start(72) first_recno(76) cur_recno(80) re_len(84) re_pad(88) rec_page(92)
b2=rd('qbase.db'); struct.pack_into('<I',b2,16,2)
struct.pack_into('<6I',b2,72,0,1,seed,rl,rp,rpg); wr('q_v2.db',b2)
# v1: QMETA30 = DBMETA30 header(56) start(56) first_recno(60) cur_recno(64) re_len(68) re_pad(72) rec_page(76)
b1=rd('qbase.db'); struct.pack_into('<I',b1,16,1)
struct.pack_into('<I',b1,28,0); struct.pack_into('<I',b1,32,0); b1[36:56]=uid
struct.pack_into('<6I',b1,56,0,1,seed,rl,rp,rpg); wr('q_v1.db',b1)
print("rewrote old-format metadata pages")
PY
)

# 2c. clean-verifying old-format upgrades: hash v5/v6 (mpool pass fixes
# last_pgno) and queue v1/v2 (queue never rewrites last_pgno).
upgrade_verify h_v5.db
upgrade_verify h_v6.db
upgrade_verify q_v1.db
upgrade_verify q_v2.db

# 2d. btree v6/v7: transform runs (version bumps to 8) but db_verify rejects
# last_pgno -- see the __db_set_lastpgno finding above.
upgrade_only b_v6.db 8
upgrade_only b_v7.db 8

echo "run_upgrade.sh: PASS"
exit 0
Loading