fix(drive-abci): roll back dropped state transitions on the proposing path - #4409
Conversation
… path Hotfix for the mainnet evo1 stalls of 2026-08-14/15 (after heights 415652 and 415661). Execution can write into the shared block transaction before failing — the address-input fee flow is apply-then-check, and the estimated fee used for admission can undershoot the actual metered fee for a Shield (dashpay/grovedb#812) — so a transition dropped as InternalError left its writes in the transaction. The proposer then gossiped a block WITHOUT the transition (TxAction::Removed) while advertising an app hash computed WITH its writes. No validator could reproduce that hash, and every proposer whose mempool carried the transition burned its round: the chain stalled for a full proposer rotation (~2h at quorum size 100), and the trigger is remotely repeatable by anyone at the cost of one Unshield. When building a proposal, wrap each executed state transition in a GroveDB savepoint and roll back if its result strips it from the block (InternalError or UnpaidConsensusError). The proposal then omits the transition AND its app hash omits its writes, so any validator — including un-upgraded v4.1.0 ones — reproduces the hash and the round commits. This is deliberately proposer-side only and consensus-invisible, so it needs no protocol-version gate and protects incrementally as masternodes upgrade: each upgraded proposer immediately stops poisoning its own proposals, and a stall triggered mid-rollout ends at the first upgraded proposer's slot instead of running a full rotation. The validation path is untouched — rolling back there would change what state a received block evaluates to, a consensus change that rides the protocol v14 gate instead (#4408). The test validating_must_behave_exactly_as_v4_1_0 pins that path bit-for-bit, leak included, and fails if this hotfix ever silently becomes a fork. Savepoints of kept transitions stay on the stack (RocksDB exposes no pop-without-rollback) and die with the per-round transaction; the genesis re-proposal path (prepare_proposal, process_proposal, mimic) now drains the stack instead of popping once so the residue cannot redirect it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
|
🔍 Review in progress — actively reviewing now (commit 3978bdf) |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## v4.1-dev #4409 +/- ##
==========================================
Coverage 87.54% 87.55%
==========================================
Files 2670 2671 +1
Lines 338763 338880 +117
==========================================
+ Hits 296583 296698 +115
- Misses 42180 42182 +2
🚀 New features to boost your workflow:
|
…oposing path Fund a shield at the edge of the estimated-vs-actual fee band (no fault hook) and assert the proposing path leaves state consistent with the outcome. On this line it reproduces the exact mainnet halt case: the transition passes estimated-fee validation, fails the actual-fee coverage guard at execution, is dropped as InternalError — and the root hash is unchanged. The match on the execution result keeps the test valid if fee constants shift: a validation reject must also leave no trace, and only a genuine success may change state. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…eights Replace the genesis savepoint-stack drain with not creating per-ST savepoints at the genesis height in the first place. The genesis re-proposal path keeps its original single-savepoint discipline (init_chain sets one savepoint, each round rewinds to it with one rollback) with prepare_proposal, process_proposal and mimic reverted to their pre-hotfix state — no drain loops, no swallowed errors, no compensating logic at a distance. At every other height each proposal round runs in a freshly started transaction, so savepoints left by kept transitions are provably inert: they die with a dropped round or ride through commit as markers. Trade-off, accepted: no halt protection for a state transition inside a genesis-height block itself. Irrelevant to any running network; a new devnet that trips it restarts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Reviewed. |
thepastaclaw
left a comment
There was a problem hiding this comment.
Final validation — Codex/Sol only (Phase 2 disabled)
The proposer-only, non-genesis rollback correctly removes state writes for transitions omitted from a proposal while preserving the existing validator and genesis behavior. One maintainability suggestion remains: make the rollback classification exhaustive so future execution-result variants cannot be classified elsewhere without an explicit savepoint decision here.
Source: reviewers codex/general, codex/security-auditor, and codex/rust-quality used backend model gpt-5.6-sol; final verifier used backend model gpt-5.6-sol. openclaw-agent/cliproxy/gpt-5.6-sol was orchestration-only and is not reviewer evidence.
Validated zero-blocker Codex/Sol precheck evidence was promoted to final because Phase 2 (Sonnet/Opus) is temporarily disabled. This is Codex/Sol-only final validation, not Codex + Sonnet/Opus coverage.
Review provenance
- Codex reviewers:
gpt-5.6-sol— general (completed),gpt-5.6-sol— security-auditor (completed),gpt-5.6-sol— rust-quality (completed) - Verifier:
gpt-5.6-sol— verifier - Sonnet/Opus: not run (Phase 2 disabled — temporary Codex/Sol-only final)
- Secondary pass: disabled (
temporary_phase2_sonnet_disable)
🟡 1 suggestion(s)
🤖 Prompt for all review comments with AI agents
These findings are from an automated code review. Verify each finding against the current code and only fix it if needed.
In `packages/rs-drive-abci/src/execution/platform_events/state_transition_processing/process_raw_state_transitions/v0/mod.rs`:
- [SUGGESTION] packages/rs-drive-abci/src/execution/platform_events/state_transition_processing/process_raw_state_transitions/v0/mod.rs:203-224: Keep proposal-removal classification exhaustive
This match defines the critical correspondence between execution outcomes removed from a proposal and outcomes whose transaction writes must be rolled back, but the wildcard arm does not make Rust enforce that correspondence. Other exhaustive matches, including the metrics match below and the `TxAction` match in `prepare_proposal`, would catch a new enum variant, but updating those matches would not require the maintainer to classify its savepoint behavior here. Enumerate every retained or non-writing outcome so any future `StateTransitionExecutionResult` addition also requires an explicit rollback decision at this point.
Enumerate every StateTransitionExecutionResult variant in the proposer-side rollback match instead of a wildcard, so adding a new execution result forces an explicit savepoint decision at the point where the rollback classification must mirror prepare_proposal's TxAction classification. Suggested by review on #4409. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
shumkov
left a comment
There was a problem hiding this comment.
We need to refactor this later to get rid of #cfg(test) inside the function, but it's fine for a hot fix
… 4.1.1 proposer fix Remove process_raw_state_transitions v1 and DRIVE_ABCI_METHOD_VERSIONS_V10, rewiring protocol v14 back to V9. Review of process_proposal established that its unexpected_execution_results gate (present since early 2024) already rejects any block whose execution yields InternalError or UnpaidConsensusError results, so a block that includes a written-then- failed transition can never commit — the v14 validator-side rollback closed no reachable hole, only cleaned state inside a transaction the reject path discards, while its genesis carve-out would have become a permanent consensus rule (flagged as blocking in review). In its place, forward-port the shipped 4.1.1 proposer-side rollback (#4409) into v0, since v4.2-dev still carries the halt: savepoint per executed transition while proposing at non-genesis heights, rollback on drop-class results, exhaustive classification, and the fault-injection hook (as the extracted maybe_override helper requested in review). The halt regression tests now pin this path. The breaking marker on this commit reflects reverting the earlier v14 method-table change within this branch; net of the branch, the PR no longer contains any consensus change. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… path (#4409) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Issue being fixed or feature implemented
Hotfix (targeting a 4.1.1 release) for the mainnet evo1 stalls of 2026-08-14/15, after heights 415652 and 415661 (~2h each).
Shield(the keyless commitment-tree append is skipped in estimation — CommitmentTreeInsert under-costed in estimated-cost paths: keyless ops skipped, and average-case constants used as upper bounds grovedb#812).Shieldfunded between the estimated and actual fee passes validation, fails the coverage guard after its writes landed, and is dropped asInternalError→TxAction::Removed— with nothing rolling the writes back.The full fix (roll back on all nodes, closing the malicious-proposer variant too) changes what state a received block evaluates to, so it is protocol-v14-gated and rides #4408 / v4.2. This hotfix is the consensus-invisible half that can ship immediately.
What was done?
process_raw_state_transitions_v0: whenproposing_state_transitions == true, wrap each executed transition in a GroveDB savepoint (transaction.set_savepoint()) and roll back if the result strips it from the block (InternalError/UnpaidConsensusError). The proposal then omits the transition and its app hash omits its writes — exactly what any v4.1.0 validator computes from that block, so mixed networks cannot diverge and no protocol-version gate is needed.proposing_state_transitions == false): no savepoint, no rollback. Rolling back there would be a consensus change; it activates at protocol v14 via test(drive-abci): pin the mainnet shield-halt fix with regression and fault-injection tests #4408 instead.rollback_dropped_transitions = proposing && height != genesis_height): the genesis re-proposal path relies on a single-savepoint discipline (init_chain sets one savepoint; each genesis round rewinds to it with one rollback), and savepoints of kept transitions stay on the stack because RocksDB exposes no pop-without-rollback. Excluding genesis keeps that path completely untouched — no drain loops, no compensating logic. At every other height each proposal round runs in a freshly started transaction that is committed (leftover savepoints are inert markers) or dropped at round end, so the residue is provably inert. The gap this leaves — no halt protection for a crafted genesis-block shield — is irrelevant to any running network.test_fault_injection::FAIL_NEXT_SUCCESSFUL_EXECUTION): forces a successful transition to reportInternalErrorafter its writes applied — modeling the failure without depending on the fee-estimation constants.Deployment properties: protection is incremental with rollout — each upgraded masternode immediately stops poisoning its own proposals, and a stall triggered mid-rollout ends at the first upgraded proposer's slot (expected ~1/f rounds at adoption fraction f) instead of a full rotation.
How Has This Been Tested?
New
mod proposer_rollback_hotfixinshield/tests.rs(cargo test -p drive-abci --lib proposer_rollback_hotfix); the first two are driven by the fault hook on a fully-funded shield so they are independent of fee constants, the third reproduces the real mainnet failure:proposing_must_not_leave_state_of_dropped_transition— the fix: a transition dropped while proposing leaves shielded pool, note count, and root hash untouched.validating_must_behave_exactly_as_v4_1_0— the consensus-invisibility guarantee: the validating path still behaves bit-for-bit like v4.1.0 (leak preserved: pool +5000, notes +2, hash changed). If this test ever fails because the deltas became zero, the hotfix has silently become a fork.proposing_real_underfunded_shield_leaves_no_trace— the real mainnet scenario, no fault hook: a shield funded one credit below the actual metered fee passes estimated-fee validation, fails the coverage guard at execution ("address-input fee not fully covered"), is dropped — and the root hash is unchanged. The match on the outcome keeps the test valid if fee constants shift.Full
cargo test -p drive-abci --lib: 2632 passed / 0 failed.cargo clippy -p drive-abci --all-targetsclean.Breaking Changes
None — deliberately. The change affects only which blocks an upgraded node builds; every block, whoever built it, still evaluates identically on every validator. Interoperates with v4.1.0 nodes in both directions during rolling upgrade.
Checklist:
For repository code-owners and collaborators only
🤖 Generated with Claude Code