feat(cketh): sweeper funding observability and the prepaid-gas gate

[mbjorkqvist]

Part of DEFI-2933 (sweeper fee funding), second of the three remaining PRs. Targets #11086.

Why

Two things are missing once funding works: an operator cannot see whether ckETH is still fully backed, and sweeping has no way to ask whether it may spend gas at all. This adds both.

The gate

check_prepaid_sweep_gas answers "does the prepaid gas cover this sweep?" against the lower bound the minter tracks from its own events — not a reading of the chain. That is what makes it fail closed: the bound errs low, so refusing can only withhold a sweep that would in fact have been paid for, which costs delay. Nothing is ever authorised that the minter's own records do not already account for.

It is a precondition on a bound, not an allowance: it neither reserves nor deducts, so a caller issuing several sweeps must subtract what it has already committed — two 6-wei sweeps both pass against a 10-wei bound. The backing invariant does not rest on the gate at all: a funding counts its whole transfer as spent once it finalizes, so every wei at the sweeper address is already covered by a burn that preceded it.

The gate has no production caller yet; S2 (DEFI-2926) is what will consult it, exactly as the fee-subaccount burn waited for the funding task. Tests exercise it directly in the meantime.

Observability

Six metrics. Burned and spent are counters, so the invariant can be alerted on directly — burned must never fall below spent. Alongside them: how far burn currently runs ahead of spend, the prepaid-gas bound, a count of fundings that failed on chain, and the age of a funding awaiting finalization.

The burn-ahead gauge is the burn of a funding still in flight plus the fees earlier fundings provisioned but never paid. Nothing draws it down — the surplus stays as ckETH backing rather than discounting a later funding — so between fundings it only ratchets up.

The age to alert on is sweeper_in_flight_funding_age_seconds: a funding that never finalizes blocks every later one, and nothing else here reveals that, since the gas-balance gauge reports what the minter has recorded rather than what is stuck. It reads 0 when no funding is outstanding, and NaN for one whose acceptance time is unknown rather than an age measured from zero. The failed-funding counter is expected to stay at zero — funding is a bare transfer to an address derived from the minter's own key, with no code to revert in — so any non-zero value wants a look.

Proposed alert conditions are recorded on DEFI-2965, which collects observability for this pipeline rather than alerting on each piece separately.

The dashboard gains a sweeper-funding section: the address, the prepaid-gas bound, the low-water mark and target it is compared against, burned and spent, the funding in flight if there is one, and the unspent burn. It renders an in-flight funding's acceptance time as a timestamp rather than an age because DashboardTemplate::from_state must stay callable outside a canister — ic_cdk::api::time() traps in unit tests — and ages belong in metrics anyway.

Stack

Merge in order; each PR targets the one above it.

#	PR	Status
—	#11060 — Read a native ETH balance via the EVM RPC canister	merged
—	#11065 — Burn ckETH from the minter's own fee subaccount	closed; folded into #11072 and #11086
—	#11072 — Add the SweeperFunding withdrawal-request variant	merged
—	#11083 — Burn-first accounting for sweeper fee funding	merged
1	#11086 — Sweeper fee-funding task, with an end-to-end test	draft
2	Sweeper funding observability and the prepaid-gas gate	this PR
3	#11097 — Adversarial end-to-end coverage of sweeper fee funding	draft

[Copilot]

Pull request overview

Adds observability and a fail-closed prepaid-gas gate for ckETH sweeper funding.

Changes:

• Caches sweeper balance observations and validates gas availability and freshness.
• Exposes funding accounting, balances, and age metrics.
• Adds dashboard reporting and tests for funding state and gate behavior.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
templates/dashboard.html	Adds sweeper-funding dashboard section.
src/sweeper.rs	Caches observed sweeper balances.
src/state/tests.rs	Updates state fixture.
src/state/sweeper_funding/tests.rs	Tests prepaid-gas gate behavior.
src/state/sweeper_funding.rs	Defines observations and fail-closed gate.
src/state.rs	Stores the volatile balance observation.
src/main.rs	Exports six sweeper-funding metrics.
src/lifecycle/init.rs	Initializes the observation cache.
src/dashboard/tests.rs	Tests sweeper dashboard output.
src/dashboard.rs	Builds sweeper dashboard data.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 1 comment.

[mbjorkqvist]

The observation about the function's semantics is correct — it is a per-request predicate against a snapshot, and two 6-wei sweeps would both pass against a 10-wei observation. I have made that explicit in the doc comment (cd36d1a), since the previous wording invited exactly this reading.

I don't think the conclusion follows, though, on two counts.

The backing invariant does not rest on this check. record_finalized_funding adds the whole transferred amount to cumulative_transferred as soon as the funding transaction finalizes, and cumulative_spent() = transferred + fees. Every wei at the sweeper address is therefore already covered by a burn that preceded the ETH moving, so drawing that balance down — even over-eagerly — cannot make cumulative spend outrun cumulative burn. What over-authorising costs is a sweep the sweeper cannot pay for: wasted signatures, nonces and outcalls, not under-backed ckETH.

There is no caller yet. Nothing in production calls check_prepaid_sweep_gas; this PR adds the observation cache and the predicate, while sweeping itself is [S2] (DEFI-2926), still in the backlog. The function takes Option<ObservedSweeperBalance> by value, so it cannot reserve anything by construction — reserving across several sweeps is the caller's job.

So rather than restructure a function whose caller does not exist, I have recorded the requirement on DEFI-2926: the sweeping task must subtract what it has already committed since the last observation, either by tracking outstanding reservations in state and including them in required or by reconciling each against its receipt. Flagging it here was useful — daily observations mean many sweeps can fall between two of them, so this would have been easy to get wrong.

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated no new comments.

Suppressed comments (1)

rs/ethereum/cketh/minter/src/sweeper.rs:62

• The cache-write behavior is not asserted by the new tests: the gate and dashboard tests inject ObservedSweeperBalance directly, while tests/sweeper_funding.rs only checks that funding lands. Removing this block—or accidentally placing it after the NotDue return—would therefore leave tests green but keep the gate at NeverObserved on the common no-funding path. Add task-level or integration coverage showing that every successful balance read refreshes the value and timestamp even when funding is not due, and that a failed read does not fabricate an observation.

    // Cached before deciding anything: sweeping consults this far more often than it changes, and
    // recording it even when no funding is due is what keeps the observation fresh.
    let observed_at_nanos = ic_cdk::api::time();
    mutate_state(|s| {
        s.last_observed_sweeper_balance = Some(ObservedSweeperBalance {
            balance: sweeper_balance,
            observed_at_nanos,
        });
    });

[Copilot]

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated no new comments.

Suppressed comments (2)

rs/ethereum/cketh/minter/src/state.rs:126

• The safety-critical upgrade behavior is not covered: existing tests exercise initial None and successful/failed reads, but none starts with Some(...), performs replay/upgrade, and verifies the observation is absent afterward. Please add that test so this volatile cache cannot later become event-sourced or upgrade-persistent unnoticed; assert the post-upgrade metrics/gate remains fail-closed until a new balance callback completes.

    /// The sweeper address' ETH balance as last read on chain, i.e. the prepaid sweep gas.
    /// Volatile cache refreshed by the funding task, deliberately not event-sourced.
    pub last_observed_sweeper_balance: Option<ObservedSweeperBalance>,

rs/ethereum/cketh/minter/src/main.rs:1135

• This gauge is not always the credit available to offset the next funding. Once a funding burn is accepted, burned_not_yet_spent() includes that new burn, while in_flight_funding().amount is explicitly earmarked and plan_funding refuses to reuse it. During that interval the metric can overstate usable credit by the entire funding amount. Either subtract the in-flight earmark when exporting the “outstanding credit” metric, or describe this as gross unspent burn and expose the offsettable credit separately.

                w.encode_gauge(
                    "cketh_minter_sweeper_funding_burned_not_yet_spent",
                    s.sweeper_funding.burned_not_yet_spent().as_f64(),
                    "ckETH burned for sweeping but not yet spent, i.e. the credit that offsets \
                     the next funding.",

[Copilot]

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 4 comments.

Suppressed comments (1)

rs/ethereum/cketh/minter/src/state/sweeper_funding.rs:135

• The gate no longer matches the contract stated in the PR: it accepts an event-derived Wei bound and can only report Insufficient, so it cannot distinguish never-observed or stale observations, and replayed funding state remains usable after an upgrade. Either implement the described non-event-sourced observation/timestamp checks (including the two-day limit) or update the PR’s requirements to this materially different design.

pub fn check_prepaid_sweep_gas(bound: Wei, required: Wei) -> Result<Wei, PrepaidGasUnavailable> {
    if bound < required {
        return Err(PrepaidGasUnavailable::Insufficient {
            available: bound,
            required,
        });

[mbjorkqvist]

Right on both counts, and the fix is the description: it still described the design this stack started with, where the funding task read the balance through the EVM RPC canister and cached it, and the gate refused on "never observed" or "stale". That was replaced by the tracked lower bound (the eth_getBalance thread on #11086), which removed the cache, the staleness limit and the age metric along with it. The description now says what the code does.

There is deliberately no balance-age metric in the new design: there is nothing to go stale, since the bound is derived from the minter's own events and cannot be older than the last funding it recorded. What can go wrong is a funding that never finalizes, which is what cketh_minter_sweeper_in_flight_funding_age_seconds is for.

[mbjorkqvist]

Same stale description, now corrected — there is no observation to show a timestamp for.

On the conflation: with a tracked bound, 0 Wei is not "never looked". It means no funding has been recorded as finalized yet, which is a fact about the minter's own history rather than a missing reading, and the row above it shows the low-water mark and target it is compared against. The distinction that mattered in the old design — "no gas" versus "never read the chain" — cannot arise when nothing reads the chain.

[mbjorkqvist]

Agreed, and fixed. That comment survived from a design where the surplus was discounted from the next burn; it now says the surplus stays as backing. The metric help text and the accounting docs already said so, which is what made the comment stand out.

[mbjorkqvist]

A real gap, thank you — the counter was only ever incremented by a test calling record_failed_funding directly, so the wiring in record_finalized_transaction could have been deleted with the suite still green.

Covered now by should_record_a_funding_whose_transaction_failed_on_chain, which drives the events rather than the accounting: it accepts a funding, creates and signs the transaction, finalizes it with a failure receipt, and asserts the counter went 0 to 1, that the bound stays at zero because nothing was delivered, that the gas the failed transaction still paid is recorded as spent, and that the burn stays put. The success paths in the same module assert the counter stays at zero.

[mbjorkqvist]

A note on how this gauge sits next to the pre-existing cketh_minter_total_unspent_tx_fees, since this is the PR that puts the funding numbers beside it.

For a user withdrawal the ckETH burned is withdrawal_amount while the ETH leaving the main address is delivered + effective, so ETH at main address − ckETH supply == Σ (charged − effective) == total_unspent_tx_fees — the figure docs/cketh.adoc describes as fees charged but in retrospect not needed. A funding behaves the same way, since it burns its full amount and never discounts a later burn, so the identity still holds with funding live.

What does not hold is reading the two side by side. For a funding that succeeds, burned_not_yet_spent grows by exactly charged − effective — the same wei total_unspent_tx_fees just counted — so summing them double counts. For one that fails they diverge instead: the whole undelivered burn stays as backing, while the unspent-fee figure only sees the fee difference. Composing them means picking the user-withdrawal part of the older counter, which nothing exposes separately today.

I would rather not add a derived over-backing gauge here for the sake of it. Flagging it so it is on the record when the alerts on DEFI-2965 are written: alert on burned versus spent, not on a sum of these two.

[mbjorkqvist]

Right on both counts, and the fix is the description: it still described the design this stack started with, where the funding task read the balance through the EVM RPC canister and cached it, and the gate refused on "never observed" or "stale". That was replaced by the tracked lower bound (the eth_getBalance thread on #11086), which removed the cache, the staleness limit and the age metric along with it. The description now says what the code does.

There is deliberately no balance-age metric in the new design: there is nothing to go stale, since the bound is derived from the minter's own events and cannot be older than the last funding it recorded. What can go wrong is a funding that never finalizes, which is what cketh_minter_sweeper_in_flight_funding_age_seconds is for.

[mbjorkqvist]

Same stale description, now corrected — there is no observation to show a timestamp for.

On the conflation: with a tracked bound, 0 Wei is not "never looked". It means no funding has been recorded as finalized yet, which is a fact about the minter's own history rather than a missing reading, and the row above it shows the low-water mark and target it is compared against. The distinction that mattered in the old design — "no gas" versus "never read the chain" — cannot arise when nothing reads the chain.

[mbjorkqvist]

Agreed, and fixed. That comment survived from a design where the surplus was discounted from the next burn; it now says the surplus stays as backing. The metric help text and the accounting docs already said so, which is what made the comment stand out.

[mbjorkqvist]

A real gap, thank you — the counter was only ever incremented by a test calling record_failed_funding directly, so the wiring in record_finalized_transaction could have been deleted with the suite still green.

Covered now by should_record_a_funding_whose_transaction_failed_on_chain, which drives the events rather than the accounting: it accepts a funding, creates and signs the transaction, finalizes it with a failure receipt, and asserts the counter went 0 to 1, that the bound stays at zero because nothing was delivered, that the gas the failed transaction still paid is recorded as spent, and that the burn stays put. The success paths in the same module assert the counter stays at zero.