Skip to content

fix(drive-abci): roll back dropped state transitions on the proposing path - #4409

Merged
shumkov merged 4 commits into
v4.1-devfrom
claude/proposer-only-rollback-411
Aug 18, 2026
Merged

fix(drive-abci): roll back dropped state transitions on the proposing path#4409
shumkov merged 4 commits into
v4.1-devfrom
claude/proposer-only-rollback-411

Conversation

@QuantumExplorer

@QuantumExplorer QuantumExplorer commented Aug 17, 2026

Copy link
Copy Markdown
Member

Issue being fixed or feature implemented

Hotfix (targeting a 4.1.1 release) for the mainnet evo1 stalls of 2026-08-14/15, after heights 415652 and 415661 (~2h each).

  • Execution can write into the shared block transaction before failing: the address-input fee flow is apply-then-check, and the admission estimate can undershoot the actual metered fee for a Shield (the keyless commitment-tree append is skipped in estimation — CommitmentTreeInsert under-costed in estimated-cost paths: keyless ops skipped, and average-case constants used as upper bounds grovedb#812).
  • A Shield funded between the estimated and actual fee passes validation, fails the coverage guard after its writes landed, and is dropped as InternalErrorTxAction::Removed — with nothing rolling the writes back.
  • The proposer gossips a block without the transition while advertising an app hash computed with its writes. No validator can reproduce that hash; every proposer whose mempool carries the transition burns its round. With a 100-validator quorum that is a full ~2h rotation, and the trigger is remotely repeatable by anyone at the cost of one Unshield.

The full fix (roll back on all nodes, closing the malicious-proposer variant too) changes what state a received block evaluates to, so it is protocol-v14-gated and rides #4408 / v4.2. This hotfix is the consensus-invisible half that can ship immediately.

What was done?

  • Proposer-side rollback in process_raw_state_transitions_v0: when proposing_state_transitions == true, wrap each executed transition in a GroveDB savepoint (transaction.set_savepoint()) and roll back if the result strips it from the block (InternalError / UnpaidConsensusError). The proposal then omits the transition and its app hash omits its writes — exactly what any v4.1.0 validator computes from that block, so mixed networks cannot diverge and no protocol-version gate is needed.
  • Validation path untouched (proposing_state_transitions == false): no savepoint, no rollback. Rolling back there would be a consensus change; it activates at protocol v14 via test(drive-abci): pin the mainnet shield-halt fix with regression and fault-injection tests #4408 instead.
  • Genesis height excluded (rollback_dropped_transitions = proposing && height != genesis_height): the genesis re-proposal path relies on a single-savepoint discipline (init_chain sets one savepoint; each genesis round rewinds to it with one rollback), and savepoints of kept transitions stay on the stack because RocksDB exposes no pop-without-rollback. Excluding genesis keeps that path completely untouched — no drain loops, no compensating logic. At every other height each proposal round runs in a freshly started transaction that is committed (leftover savepoints are inert markers) or dropped at round end, so the residue is provably inert. The gap this leaves — no halt protection for a crafted genesis-block shield — is irrelevant to any running network.
  • Test-only fault hook (test_fault_injection::FAIL_NEXT_SUCCESSFUL_EXECUTION): forces a successful transition to report InternalError after its writes applied — modeling the failure without depending on the fee-estimation constants.

Deployment properties: protection is incremental with rollout — each upgraded masternode immediately stops poisoning its own proposals, and a stall triggered mid-rollout ends at the first upgraded proposer's slot (expected ~1/f rounds at adoption fraction f) instead of a full rotation.

How Has This Been Tested?

New mod proposer_rollback_hotfix in shield/tests.rs (cargo test -p drive-abci --lib proposer_rollback_hotfix); the first two are driven by the fault hook on a fully-funded shield so they are independent of fee constants, the third reproduces the real mainnet failure:

  • proposing_must_not_leave_state_of_dropped_transition — the fix: a transition dropped while proposing leaves shielded pool, note count, and root hash untouched.
  • validating_must_behave_exactly_as_v4_1_0 — the consensus-invisibility guarantee: the validating path still behaves bit-for-bit like v4.1.0 (leak preserved: pool +5000, notes +2, hash changed). If this test ever fails because the deltas became zero, the hotfix has silently become a fork.
  • proposing_real_underfunded_shield_leaves_no_trace — the real mainnet scenario, no fault hook: a shield funded one credit below the actual metered fee passes estimated-fee validation, fails the coverage guard at execution ("address-input fee not fully covered"), is dropped — and the root hash is unchanged. The match on the outcome keeps the test valid if fee constants shift.

Full cargo test -p drive-abci --lib: 2632 passed / 0 failed. cargo clippy -p drive-abci --all-targets clean.

Breaking Changes

None — deliberately. The change affects only which blocks an upgraded node builds; every block, whoever built it, still evaluates identically on every validator. Interoperates with v4.1.0 nodes in both directions during rolling upgrade.

Checklist:

  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have added or updated relevant unit/integration/functional/e2e tests
  • I have added "!" to the title and described breaking changes in the corresponding section if my code contains any
  • I have made corresponding changes to the documentation if needed

For repository code-owners and collaborators only

  • I have assigned this pull request to a milestone

🤖 Generated with Claude Code

… path

Hotfix for the mainnet evo1 stalls of 2026-08-14/15 (after heights
415652 and 415661). Execution can write into the shared block
transaction before failing — the address-input fee flow is
apply-then-check, and the estimated fee used for admission can
undershoot the actual metered fee for a Shield (dashpay/grovedb#812) —
so a transition dropped as InternalError left its writes in the
transaction. The proposer then gossiped a block WITHOUT the transition
(TxAction::Removed) while advertising an app hash computed WITH its
writes. No validator could reproduce that hash, and every proposer
whose mempool carried the transition burned its round: the chain
stalled for a full proposer rotation (~2h at quorum size 100), and the
trigger is remotely repeatable by anyone at the cost of one Unshield.

When building a proposal, wrap each executed state transition in a
GroveDB savepoint and roll back if its result strips it from the block
(InternalError or UnpaidConsensusError). The proposal then omits the
transition AND its app hash omits its writes, so any validator —
including un-upgraded v4.1.0 ones — reproduces the hash and the round
commits.

This is deliberately proposer-side only and consensus-invisible, so it
needs no protocol-version gate and protects incrementally as
masternodes upgrade: each upgraded proposer immediately stops poisoning
its own proposals, and a stall triggered mid-rollout ends at the first
upgraded proposer's slot instead of running a full rotation. The
validation path is untouched — rolling back there would change what
state a received block evaluates to, a consensus change that rides the
protocol v14 gate instead (#4408). The test
validating_must_behave_exactly_as_v4_1_0 pins that path bit-for-bit,
leak included, and fails if this hotfix ever silently becomes a fork.

Savepoints of kept transitions stay on the stack (RocksDB exposes no
pop-without-rollback) and die with the per-round transaction; the
genesis re-proposal path (prepare_proposal, process_proposal, mimic)
now drains the stack instead of popping once so the residue cannot
redirect it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@github-actions github-actions Bot added this to the v4.1.0 milestone Aug 17, 2026
@coderabbitai

coderabbitai Bot commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 0de1e12f-4225-4e6d-9290-6a9653c7f3c5

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Comment @coderabbitai help to get the list of available commands.

@thepastaclaw

thepastaclaw commented Aug 17, 2026

Copy link
Copy Markdown
Collaborator

🔍 Review in progress — actively reviewing now (commit 3978bdf)

@codecov

codecov Bot commented Aug 17, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 92.30769% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 87.55%. Comparing base (bfc8024) to head (c95e826).

Files with missing lines Patch % Lines
...processing/process_raw_state_transitions/v0/mod.rs 92.30% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           v4.1-dev    #4409    +/-   ##
==========================================
  Coverage     87.54%   87.55%            
==========================================
  Files          2670     2671     +1     
  Lines        338763   338880   +117     
==========================================
+ Hits         296583   296698   +115     
- Misses        42180    42182     +2     
Components Coverage Δ
dpp 88.49% <ø> (ø)
drive 86.33% <ø> (ø)
drive-abci 89.57% <92.30%> (+<0.01%) ⬆️
sdk ∅ <ø> (∅)
dapi-client ∅ <ø> (∅)
platform-version ∅ <ø> (∅)
platform-value 92.88% <ø> (ø)
platform-wallet ∅ <ø> (∅)
drive-proof-verifier 49.79% <ø> (ø)
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

QuantumExplorer and others added 2 commits August 17, 2026 18:39
…oposing path

Fund a shield at the edge of the estimated-vs-actual fee band (no fault
hook) and assert the proposing path leaves state consistent with the
outcome. On this line it reproduces the exact mainnet halt case: the
transition passes estimated-fee validation, fails the actual-fee
coverage guard at execution, is dropped as InternalError — and the root
hash is unchanged. The match on the execution result keeps the test
valid if fee constants shift: a validation reject must also leave no
trace, and only a genuine success may change state.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…eights

Replace the genesis savepoint-stack drain with not creating per-ST
savepoints at the genesis height in the first place. The genesis
re-proposal path keeps its original single-savepoint discipline
(init_chain sets one savepoint, each round rewinds to it with one
rollback) with prepare_proposal, process_proposal and mimic reverted to
their pre-hotfix state — no drain loops, no swallowed errors, no
compensating logic at a distance. At every other height each proposal
round runs in a freshly started transaction, so savepoints left by kept
transitions are provably inert: they die with a dropped round or ride
through commit as markers.

Trade-off, accepted: no halt protection for a state transition inside a
genesis-height block itself. Irrelevant to any running network; a new
devnet that trips it restarts.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@QuantumExplorer

Copy link
Copy Markdown
Member Author

Reviewed.

@thepastaclaw thepastaclaw left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Final validation — Codex/Sol only (Phase 2 disabled)

The proposer-only, non-genesis rollback correctly removes state writes for transitions omitted from a proposal while preserving the existing validator and genesis behavior. One maintainability suggestion remains: make the rollback classification exhaustive so future execution-result variants cannot be classified elsewhere without an explicit savepoint decision here.
Source: reviewers codex/general, codex/security-auditor, and codex/rust-quality used backend model gpt-5.6-sol; final verifier used backend model gpt-5.6-sol. openclaw-agent/cliproxy/gpt-5.6-sol was orchestration-only and is not reviewer evidence.

Validated zero-blocker Codex/Sol precheck evidence was promoted to final because Phase 2 (Sonnet/Opus) is temporarily disabled. This is Codex/Sol-only final validation, not Codex + Sonnet/Opus coverage.

Review provenance

  • Codex reviewers: gpt-5.6-sol — general (completed), gpt-5.6-sol — security-auditor (completed), gpt-5.6-sol — rust-quality (completed)
  • Verifier: gpt-5.6-sol — verifier
  • Sonnet/Opus: not run (Phase 2 disabled — temporary Codex/Sol-only final)
  • Secondary pass: disabled (temporary_phase2_sonnet_disable)

🟡 1 suggestion(s)

🤖 Prompt for all review comments with AI agents
These findings are from an automated code review. Verify each finding against the current code and only fix it if needed.

In `packages/rs-drive-abci/src/execution/platform_events/state_transition_processing/process_raw_state_transitions/v0/mod.rs`:
- [SUGGESTION] packages/rs-drive-abci/src/execution/platform_events/state_transition_processing/process_raw_state_transitions/v0/mod.rs:203-224: Keep proposal-removal classification exhaustive
  This match defines the critical correspondence between execution outcomes removed from a proposal and outcomes whose transaction writes must be rolled back, but the wildcard arm does not make Rust enforce that correspondence. Other exhaustive matches, including the metrics match below and the `TxAction` match in `prepare_proposal`, would catch a new enum variant, but updating those matches would not require the maintainer to classify its savepoint behavior here. Enumerate every retained or non-writing outcome so any future `StateTransitionExecutionResult` addition also requires an explicit rollback decision at this point.

Enumerate every StateTransitionExecutionResult variant in the
proposer-side rollback match instead of a wildcard, so adding a new
execution result forces an explicit savepoint decision at the point
where the rollback classification must mirror prepare_proposal's
TxAction classification.

Suggested by review on #4409.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

@shumkov shumkov left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to refactor this later to get rid of #cfg(test) inside the function, but it's fine for a hot fix

@shumkov
shumkov merged commit 1794492 into v4.1-dev Aug 18, 2026
21 of 22 checks passed
@shumkov
shumkov deleted the claude/proposer-only-rollback-411 branch August 18, 2026 00:56
QuantumExplorer added a commit that referenced this pull request Aug 19, 2026
… 4.1.1 proposer fix

Remove process_raw_state_transitions v1 and DRIVE_ABCI_METHOD_VERSIONS_V10,
rewiring protocol v14 back to V9. Review of process_proposal established
that its unexpected_execution_results gate (present since early 2024)
already rejects any block whose execution yields InternalError or
UnpaidConsensusError results, so a block that includes a written-then-
failed transition can never commit — the v14 validator-side rollback
closed no reachable hole, only cleaned state inside a transaction the
reject path discards, while its genesis carve-out would have become a
permanent consensus rule (flagged as blocking in review).

In its place, forward-port the shipped 4.1.1 proposer-side rollback
(#4409) into v0, since v4.2-dev still carries the halt: savepoint per
executed transition while proposing at non-genesis heights, rollback on
drop-class results, exhaustive classification, and the fault-injection
hook (as the extracted maybe_override helper requested in review). The
halt regression tests now pin this path.

The breaking marker on this commit reflects reverting the earlier
v14 method-table change within this branch; net of the branch, the PR
no longer contains any consensus change.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
QuantumExplorer added a commit that referenced this pull request Aug 19, 2026
… path (#4409)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants