test(cketh): adversarial end-to-end coverage of sweeper fee funding - #11097
test(cketh): adversarial end-to-end coverage of sweeper fee funding#11097mbjorkqvist wants to merge 26 commits into
Conversation
There was a problem hiding this comment.
Pull request overview
Adds adversarial end-to-end coverage ensuring sweeper fee funding fails safely.
Changes:
- Tests empty fee accounts, sufficient sweeper balances, and reverted funding transactions.
- Extends the live harness with dashboard, balance, code, status, and finalization helpers.
- Adds a dedicated long-running Bazel target.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
rs/ethereum/cketh/test_utils/src/sweeper_funding.rs |
Expands live-test utilities. |
rs/ethereum/cketh/minter/tests/sweeper_funding.rs |
Uses dashboard-derived sweeper address. |
rs/ethereum/cketh/minter/tests/sweeper_funding_hardening.rs |
Adds three adversarial scenarios. |
rs/ethereum/cketh/minter/BUILD.bazel |
Registers the long-running test target. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| let burn_index = setup | ||
| .in_flight_funding_burn_index() | ||
| .expect("the funding must be in flight once its burn has happened"); |
There was a problem hiding this comment.
Correct, and well spotted — fixed in 2096a99.
The ordering is as you describe: fund_sweeper_address awaits the ledger burn and records AcceptedSweeperFundingRequest only after that call returns, while await_burn watches the ledger's total supply through an external query. So there is a real window in which the supply has dropped and the dashboard still shows no in-flight funding, and reading the index there would panic on an empty row.
It now polls, with the deadline as a named constant because it has to sit between two bounds rather than merely being long enough: generous for an inter-canister hop, but well under the roughly six minutes before the transaction finalizes and clears that row again. Polling indefinitely would have traded this race for the opposite one.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.
Suppressed comments (1)
rs/ethereum/cketh/minter/tests/sweeper_funding_hardening.rs:83
- The no-burn baseline is captured after
upgrade_minter(), which starts the zero-delay funding timer on this live PocketIC instance. The task can therefore complete its ledger burn before these queries run, causing both “unchanged” assertions to compare against an already-debited state. Capture the supply and fee-account balance before re-arming the timer so the test cannot miss the behavior it is intended to reject.
let supply_before = setup.cketh_total_supply();
let fee_account_before = setup.cketh_balance_of(setup.fee_account());
| // zero-delay timer, so a minter that wrongly funded could burn before these queries returned | ||
| // and both assertions below would then compare against an already-debited state — passing | ||
| // precisely when the behaviour they reject had happened. | ||
| let supply_before = setup.cketh_total_supply(); |
There was a problem hiding this comment.
Replying to the suppressed comment about this baseline being captured after upgrade_minter() (no comment node to thread onto, hence a fresh one here).
Right, and a fair catch — fixed in 532989e.
Worth being precise about the failure mode, because it is not a flake in the current code: with a correct minter the sweeper is above the low-water mark, no funding is due, and nothing is burned, so the ordering never bites. It matters for a broken minter — if the post-upgrade check wrongly funded, its burn could land before those two queries returned, and then both "unchanged" assertions would compare against an already-debited state. The test would go green at precisely the moment it was supposed to fail.
The baselines are now read before upgrade_minter(), which is the one point where nothing can yet have happened: the fee account is funded, the sweeper is arranged, and the next scheduled check is a whole interval away. The revert test below already did it in that order — I fixed it there when addressing the in-flight race and did not carry the reasoning back up.
I left the first test as it is: its fee account is empty, so no burn can succeed at all, and its log assertion proves the task ran and declined at the burn rather than never running.
…ity#11060) Part of [DEFI-2933](https://dfinity.atlassian.net/browse/DEFI-2933) (sweeper fee funding), first of a seven-PR stack. ## Why Funding the sweeper address with gas requires knowing how much gas it already holds. The EVM RPC canister exposes no endpoint for a native ETH balance, and its Rust client offers no getter for one, so the minter currently has no way to ask. ## What Reads the balance through the EVM RPC canister's generic JSON-RPC passthrough, which forwards a payload to every provider and agrees on one answer under the configured consensus strategy. That strategy is a threshold of the providers — 3 of 4 on mainnet, 2 of 4 on Sepolia — and it is the only agreement accepted: there is no client-side reduction, so a result the canister reports as inconsistent stays an error rather than being resolved by picking a winner. Because the canister deserializes each response's `result` field, what the minter receives is the quantity itself rather than any surrounding JSON. It is therefore decoded exactly: quotes, padding, leading zeros and sign characters are the provider's own malformation and are rejected rather than repaired. A failed read is an error, never a zero. This is the decision the rest of the stack depends on: confusing "could not read the balance" with "no gas left" would burn ckETH to top up an address that is already funded, which is pure loss. The request builder and the result decoder are pure functions so both sides of that guarantee are pinned directly, including a test asserting that no error input can decode to a zero balance. The route was also proven end to end against a live EVM RPC canister and a local anvil node, reaching 3-of-4 consensus at both `latest` and `finalized`. ## Stack Merge in order; each PR targets the one above it. | # | PR | Status | |---|----|--------| | 1 | Read a native ETH balance via the EVM RPC canister | **this PR** | | 2 | dfinity#11065 — Burn ckETH from the minter's own fee subaccount | ready for review | | 3 | dfinity#11072 — Add the SweeperFunding withdrawal-request variant | ready for review | | 4 | dfinity#11083 — Burn-first accounting for sweeper fee funding | ready for review | | 5 | dfinity#11086 — Sweeper fee-funding task, with an end-to-end test | Copilot re-review pending, CI green incl. long tests | | 6 | dfinity#11094 — Sweeper funding observability and the prepaid-gas gate | open | | 7 | dfinity#11097 — Adversarial end-to-end coverage of sweeper fee funding | open | [DEFI-2933]: https://dfinity.atlassian.net/browse/DEFI-2933?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
4d829c1 to
e5adda2
Compare
532989e to
b48a599
Compare
b633db9 to
9d7cb6d
Compare
b48a599 to
df115ab
Compare
9d7cb6d to
c062a60
Compare
df115ab to
fc087d2
Compare
c062a60 to
5487cb6
Compare
fc087d2 to
c4eefd7
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.
Suppressed comments (2)
rs/ethereum/cketh/minter/tests/sweeper_funding_hardening.rs:196
- Checking only that the surplus is nonzero does not verify the stated failed-transaction accounting. If finalization accidentally recorded a zero transaction fee, the entire burn would appear as surplus and this test would still pass. Assert that
sweeper-eth-spentis nonzero after finalization so the end-to-end path proves failed transactions still charge and record gas.
assert_ne!(
surplus, "0 Wei",
"the unreimbursed burn must still show as burned but unspent, got {surplus}"
);
rs/ethereum/cketh/minter/tests/sweeper_funding_hardening.rs:123
- This description is incorrect for a failed Ethereum transaction: the transfer value is reverted, but the sender still loses the effective gas fee. Clarify that only the value fails to reach the sweeper; otherwise this contradicts the accounting exercised below.
This issue also appears on line 193 of the same file.
/// A funding transaction that fails on chain is never reimbursed. The ETH never leaves the main
/// address and the ckETH stays burned, so the burn ends up as extra backing.
…ty#11072) Part of [DEFI-2933](https://dfinity.atlassian.net/browse/DEFI-2933) (sweeper fee funding). Now targets `master`: dfinity#11060 has merged, and dfinity#11065 is being closed with its contents folded into the PRs that use them — the fee-subaccount constant into this one, the burn helper into dfinity#11086 alongside its caller. ## Why Sweeper fee funding is mechanically an ordinary ckETH withdrawal — same nonce sequence, same threshold-ECDSA signing, same fee-bumped resubmission — so it becomes a third `WithdrawalRequest` variant rather than a parallel pipeline. It differs in exactly one respect, and that difference is what the whole feature turns on: the ckETH burned for funding is **never** re-minted. A funding request must therefore never reach the reimbursement machinery. ## What Three places enforce that, all of which would otherwise fail only at runtime: - `maybe_reimburse` is the double-minting guard, and `record_reimbursement_request` asserts membership has been cleared before minting. Funding is kept out of the set on insert, and the corresponding assertion on removal is made conditional. Both are production assertions, so a missed branch traps the canister. - The conversion to a reimbursement index becomes fallible, deliberately: a fallible conversion makes the compiler prove at every call site that funding cannot produce an index, rather than relying on a panicking arm that traps if a site is missed. The two callers construct their index inside the reimbursable arms instead. - Nothing is recorded to pay back if a funding transaction finalizes with a failure receipt — an outcome that requires code at the destination, so it cannot arise for a bare transfer to the sweeper, and is logged as unexpected rather than handled. The ckETH stays burned while no ETH reaches the sweeper — only the gas the failed transaction still paid leaves the main address — so the rest of the burn over-backs ckETH instead of anything being lost, and neither a reimbursement path nor a second event type is needed. Everything else follows ckETH: the 21'000 gas limit of a plain value transfer to a code-less address, a resubmission strategy ceilinged at the burned amount — so a climbing gas price shrinks the ETH delivered to the sweeper rather than spending more than was burned — and a fee carved out of that same amount, so balance accounting needs no change. Since reimbursement is the only difference, the variant carries an `EthWithdrawalRequest` — the same payload a user's ckETH withdrawal carries — rather than a near-copy of it, and the spec drops the one piece of funding accounting that had no counterpart in the withdrawal flow: burned-but-unspent amounts are no longer credited against the next funding's burn. They stay as backing, just like the unspent gas a user's withdrawal leaves behind. Funding appears in the withdrawal status endpoint, the dashboard and the event log rather than being hidden: it moves ckETH-denominated value and is a public, auditable action. A user query never matches one, since the sender is the minter itself. The new event takes tag 27; 26 went to `AutomaticDepositReceived` on master while this branch was open, and the tags are the durable CBOR encoding, so they cannot collide. ## Stack Merge in order; each PR targets the one above it. | # | PR | Status | |---|----|--------| | — | dfinity#11060 — Read a native ETH balance via the EVM RPC canister | merged | | — | dfinity#11065 — Burn ckETH from the minter's own fee subaccount | closed; folded into dfinity#11072 and dfinity#11086 | | 1 | Add the SweeperFunding withdrawal-request variant | **this PR** | | 2 | dfinity#11083 — Burn-first accounting for sweeper fee funding | ready for review | | 3 | dfinity#11086 — Sweeper fee-funding task, with an end-to-end test | draft | | 4 | dfinity#11094 — Sweeper funding observability and the prepaid-gas gate | draft | | 5 | dfinity#11097 — Adversarial end-to-end coverage of sweeper fee funding | draft | [DEFI-2933]: https://dfinity.atlassian.net/browse/DEFI-2933?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
5487cb6 to
572e5a0
Compare
The burn index was read once, immediately after the burn was observed. The minter records the funding only after the ledger call it awaited returns, and the burn is observed through the ledger's total supply — an external query — so there is a window where the supply has dropped and the dashboard still shows no in-flight funding. Reading there panicked on an empty row. Polled instead, with a deadline that has to sit between two bounds: generous enough for an inter-canister hop, and well under the six minutes before the transaction finalizes and clears the row again. Also drops a doc line on `await_funding_finalized` that described a main address accessor, copied from another method.
The baseline was read after `upgrade_minter()`, which re-arms a zero-delay funding check. With a correct minter nothing is due and the ordering never bites, but a minter that wrongly funded could burn before those two queries returned — and then "supply unchanged" and "fee account unchanged" would both compare against an already-debited state. The test would pass at exactly the moment it was meant to fail. Read before the upgrade instead, the one point where nothing can have happened yet: the fee account is funded, the sweeper is arranged, and the next scheduled check is a whole interval away. The revert test below already did it in this order.
The dedicated Failed status was reverted after review: it meant adding a variant to retrieve_eth_status' return type, breaking existing clients, to describe a state mainnet cannot reach — a bare transfer to a code-less address the minter's own key controls has nothing to revert in. This test reaches it anyway, by placing reverting code at the sweeper, so it now expects the pending-reimbursement status and says why that label is tolerated. What it asserts either way is the invariant: no reimbursement, and the burn stays as prepaid gas. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The assertions hold unchanged now that burned-but-unspent funds are no longer reused, but the prose still called the burn prepaid gas. Nothing reaches the sweeper when the transfer fails, so the ETH stays at the main address and the burn simply over-backs ckETH.
0bcd43c to
7a048b0
Compare
fecb8f4 to
8b29237
Compare
…-hardening Brings in the balance bound that replaces the on-chain read, and the harness that buys withdrawal-timer ticks instead of waiting them out. The hardening tests are adapted to both in the commits that follow. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The three live hardening scenarios spent almost all their time asleep: two watched an eight-minute wall-clock window to show the minter does *not* act, and the third waited out two 6-minute withdrawal timers. They now buy those ticks by pushing the instance's clock forward, and come out stronger for it — each watches across three withdrawal-timer intervals the minter genuinely runs, where before they watched a window barely longer than one. The topped-up-sweeper scenario needed rethinking as well as speeding up. It used to place ETH at the sweeper address on anvil, which worked only while the minter read that balance on chain. The minter now decides from a bound it derives from its own recorded fundings, so ETH arriving behind its back moves nothing — which is the point of the bound. The scenario therefore arranges the balance the only way that counts: it lets a real funding land and finalize, then re-arms the timers and shows the next check declines. Affordable only because a tick now costs seconds. Dropping the on-chain arrangement leaves the harness' `set_eth_balance` and `mine` with no callers, so both go. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The constructor moved here from the branch below, where nothing called it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
| # Three live scenarios, two of which watch across several withdrawal-timer ticks to show the | ||
| # minter does *not* act. Those ticks are bought by advancing the instance's clock, so the | ||
| # minutes of minter time they cover cost seconds of wall clock. | ||
| size = "medium", |
There was a problem hiding this comment.
Correct, and the description was the stale half — the tag went away when the test stopped waiting out the withdrawal timer and started buying ticks with advance_time. The whole target now runs in about 50 seconds, so running it in ordinary CI is intended, and the description says so instead of claiming long_test.
Keeps this branch's `await_sweeper_address` name and dashboard helpers, and takes the shared deadline over the per-call-site ones — including the in-flight wait, whose own constant held the same 60 seconds.
Two of Copilot's observations, both fair. The surplus assertion only checked that some burn remained, which would also hold if finalization had recorded no gas at all — it now reads the rows as numbers and pins spend against the burn. And "never reimbursed" was established by looking once at finalization, though reimbursement runs on its own timer; the test now buys a tick longer than that interval and looks again. Also corrects the doc comment: a failed transaction does move ETH out of the main address — its gas. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
| # Three live scenarios, two of which watch across several withdrawal-timer ticks to show the | ||
| # minter does *not* act. Those ticks are bought by advancing the instance's clock, so the | ||
| # minutes of minter time they cover cost seconds of wall clock. | ||
| size = "medium", |
There was a problem hiding this comment.
Correct, and the description was the stale half — the tag went away when the test stopped waiting out the withdrawal timer and started buying ticks with advance_time. The whole target now runs in about 50 seconds, so running it in ordinary CI is intended, and the description says so instead of claiming long_test.
|
|
||
| /// A funding transaction that fails on chain is never reimbursed. No ETH reaches the sweeper — only | ||
| /// the gas the failed transaction still pays leaves the main address — and the ckETH stays burned, | ||
| /// so the burn minus that gas ends up as extra backing. |
There was a problem hiding this comment.
Replying to the suppressed comment on this doc comment, which has no node to thread onto.
You were right and it is fixed: a failed transaction does move ETH out of the main address — its gas. The comment now says no ETH reaches the sweeper, that only the gas leaves, and that the burn minus that gas is what stays as backing. The same overstatement was in the funding task's log line and in #11086's description; both are corrected there.
| ); | ||
| // Read as numbers rather than asserted non-zero: "some burn remains" would also hold if | ||
| // finalization had recorded no gas at all, which is the other half of what this test claims. | ||
| let spent = wei_row(&setup, "sweeper-eth-spent"); |
There was a problem hiding this comment.
Two suppressed comments landed on the assertions here, both fair, both now acted on.
The first: assert_ne!(surplus, "0 Wei") also passes if finalization recorded no gas at all, which is half of what this test claims. The rows are read as numbers now — spend must be non-zero, and the surplus must equal the burn minus that spend, so a zero-fee regression fails here.
The second: "never reimbursed" was established by looking once at finalization, while reimbursement runs on its own three-minute timer. The test now buys a tick longer than that interval and asserts the supply is still reduced, so a funding wrongly queued for reimbursement is caught rather than minted after the test ends. Costs about four seconds.
The failed-transaction test seeded the fee account with a mint; it now deposits, and waits for the scrape that credits it before taking its baseline.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The three adversarial tests move onto the wrapper and into `deposit_from_cex` alongside the happy path, so the feature has one live target rather than three. The helpers only they need — the dashboard readers, anvil's `setCode`, the status query, the bounded-negative watcher — come with them, onto the shared fixture. That target now runs four live tests one at a time, about four minutes, so it is `size = "large"`.
Checkpoint of the merged harness on this branch; it lands in the first PR of the stack, which is where the two harnesses were.
# Conflicts: # rs/ethereum/cketh/minter/tests/deposit_from_cex.rs # rs/ethereum/cketh/test_utils/src/live.rs
Part of DEFI-2933 (sweeper fee funding), last of the three remaining PRs. Targets #11094.
Why
The stack's existing live test proves funding works. This one proves it fails safe. Each test drives a distinct way funding can go wrong through the real pipeline — real ckETH ledger, real EVM RPC canister, local anvil, threshold-ECDSA signature — and asserts the invariant the feature exists to protect: ETH never moves ahead of the ckETH burned to pay for it.
What
Three adversarial scenarios:
No live fee-spike test: that ceiling is pinned exactly by unit tests, and reproducing it here would mean driving anvil's base fee up over several ticks for little extra signal.
What it found
Waiting for finalization — rather than asserting mid-flight, as the earlier live test did — surfaced a production defect that would trap the withdrawal timer permanently. The fix ships in #11086, where it belongs; this PR is what made it visible.
Harness work
These tests need to arrange a chain and then read the minter's own view of it, so the harness gains an empty-fee-account fixture, anvil's
setCode/getCode, a withdrawal-status query and dashboard row readers. Two larger changes they depend on ship in #11086, where the happy-path test needs them too: a genuineReceivedEthlog to scrape, so the minter's ETH balance counter is backed the way it is in production rather than credited behind its back, and ticks bought withadvance_timeinstead of waited out.The whole target runs in about 50 seconds, so it is
size = "medium"in ordinary CI with nolong_testtag and no explicit timeout.Stack
Merge in order; each PR targets the one above it.