Motivation
#217 reworks reorg handling so that a crash during stale-branch cleanup is recoverable: the common ancestor is persisted before stale history deletion, and startup stale cleanup completes the job on restart. That recovery path currently has no end-to-end test.
The recoverable-restart case is straightforward to cover externally — stop electrs for the whole reorg, restart it on its preserved DB, and verify the index converges (tip, tx status, history, balances, UTXOs over REST and Electrum). What an external test cannot do is kill electrs during stale cleanup: on regtest, cleaning up a few stale blocks takes milliseconds, so a kill from outside the process cannot land in that window reliably, and a timing-based test would be flaky by construction.
Proposal
Add a narrowly scoped, test-only failpoint (pause or panic hook) at one or more of:
- after the rollback common ancestor is persisted, before stale
history deletion begins;
- inside
process_stale(), after some stale rows have been removed;
- after stale deletion is flushed, before replacement indexing is published.
Requirements:
- Inert in normal builds/runs — behind a cargo feature (e.g.
failpoints, possibly using the fail crate) and/or an env var such as ELECTRS_FAILPOINT=<name>; zero behavior change when unset.
- Observable — emit a distinct log line (and/or touch a sentinel file in the DB dir) when the failpoint is reached, so a test kills the process only after the target point is provably hit. No sleeps, no probabilistic timing.
- Simple semantics — either block until killed (
pause) or panic at the named point.
How it would be used
An integration test would: build with the feature enabled, index a chain, trigger a reorg with the failpoint armed, wait for the failpoint signal, kill the process, restart clean, and verify the index fully recovers to the daemon's best chain.
Out of scope
Marker permutations (e.g. txstore-complete/history-incomplete, out-of-order block completion) remain better suited to Rust-level Store/Indexer unit tests than to black-box failpoint testing.
Motivation
#217 reworks reorg handling so that a crash during stale-branch cleanup is recoverable: the common ancestor is persisted before stale
historydeletion, and startup stale cleanup completes the job on restart. That recovery path currently has no end-to-end test.The recoverable-restart case is straightforward to cover externally — stop electrs for the whole reorg, restart it on its preserved DB, and verify the index converges (tip, tx status, history, balances, UTXOs over REST and Electrum). What an external test cannot do is kill electrs during stale cleanup: on regtest, cleaning up a few stale blocks takes milliseconds, so a kill from outside the process cannot land in that window reliably, and a timing-based test would be flaky by construction.
Proposal
Add a narrowly scoped, test-only failpoint (pause or panic hook) at one or more of:
historydeletion begins;process_stale(), after some stale rows have been removed;Requirements:
failpoints, possibly using thefailcrate) and/or an env var such asELECTRS_FAILPOINT=<name>; zero behavior change when unset.pause) or panic at the named point.How it would be used
An integration test would: build with the feature enabled, index a chain, trigger a reorg with the failpoint armed, wait for the failpoint signal, kill the process, restart clean, and verify the index fully recovers to the daemon's best chain.
Out of scope
Marker permutations (e.g. txstore-complete/history-incomplete, out-of-order block completion) remain better suited to Rust-level
Store/Indexerunit tests than to black-box failpoint testing.