Skip to content

Optimize conflict-heavy EVM transfer block execution#3376

Open
codchen wants to merge 13 commits intomainfrom
tony/scheduler-conflict-log
Open

Optimize conflict-heavy EVM transfer block execution#3376
codchen wants to merge 13 commits intomainfrom
tony/scheduler-conflict-log

Conversation

@codchen
Copy link
Copy Markdown
Collaborator

@codchen codchen commented May 4, 2026

Summary

  • Adds local EVM stress tooling for the mainnet-shaped workload: many brand-new EVM senders submitting simple transfers to the same recipient.
  • Adds OCC scheduler conflict logging for both sei-cosmos/tasks/scheduler.go and giga/deps/tasks/scheduler.go, including the store key names and aggregate conflict counts per block.
  • Adds x/evm EndBlock latency logs for the major sections of end-block processing, plus a total wall-time log.
  • Reduces EVM association overhead by reusing an already-preprocessed derived sender in DeliverTx, reusing an empty cast auth account when safe, and skipping direct-cast self migration.
  • Detects large simple EVM transfer blocks that all target the same recipient and routes them through the sequential execution path instead of OCC. This now applies to both the regular V2 path and the Giga path.

Stress Results

  • Before the optimization, the local stress workload produced large blocks taking roughly 1.5s-3.3s under OCC, with conflicts on acc/globalAccountNumber and the shared recipient bank keys.
  • After the changes, the same local stress script produced roughly 0.30s-0.35s total execution time for ~1k tx blocks, with the conflict-heavy transfer shape avoiding OCC retry churn.

Test Plan

  • go test ./app ./utils/helpers ./app/ante ./x/evm/ante ./x/evm/types
  • Ran ./scripts/evm_stress.sh locally and confirmed large same-recipient transfer blocks execute around 300ms-350ms
  • Confirmed OCC conflict logs and EVM EndBlock latency logs are emitted for the intended paths

@github-actions
Copy link
Copy Markdown

github-actions Bot commented May 4, 2026

The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).

BuildFormatLintBreakingUpdated (UTC)
✅ passed✅ passed✅ passed✅ passedMay 6, 2026, 3:52 AM

@codecov
Copy link
Copy Markdown

codecov Bot commented May 4, 2026

Codecov Report

❌ Patch coverage is 38.42105% with 117 lines in your changes missing coverage. Please review.
✅ Project coverage is 59.07%. Comparing base (fad6d58) to head (9a4abc1).

Files with missing lines Patch % Lines
scripts/evm_stress/main.go 0.00% 73 Missing ⚠️
giga/deps/tasks/scheduler.go 0.00% 24 Missing ⚠️
app/app.go 50.00% 9 Missing and 2 partials ⚠️
app/ante/evm_delivertx.go 45.45% 3 Missing and 3 partials ⚠️
sei-cosmos/tasks/scheduler.go 87.50% 1 Missing and 2 partials ⚠️
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #3376      +/-   ##
==========================================
- Coverage   59.10%   59.07%   -0.03%     
==========================================
  Files        2101     2102       +1     
  Lines      173162   173337     +175     
==========================================
+ Hits       102344   102402      +58     
- Misses      61943    62053     +110     
- Partials     8875     8882       +7     
Flag Coverage Δ
sei-chain-pr 56.86% <38.42%> (?)
sei-db 70.41% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
sei-cosmos/baseapp/abci.go 63.00% <100.00%> (+1.07%) ⬆️
sei-cosmos/baseapp/baseapp.go 75.80% <ø> (ø)
sei-cosmos/store/multiversion/store.go 91.96% <100.00%> (+0.29%) ⬆️
utils/helpers/associate.go 72.72% <100.00%> (+7.34%) ⬆️
sei-cosmos/tasks/scheduler.go 94.88% <87.50%> (-0.74%) ⬇️
app/ante/evm_delivertx.go 67.74% <45.45%> (-4.81%) ⬇️
app/app.go 69.55% <50.00%> (-0.30%) ⬇️
giga/deps/tasks/scheduler.go 84.26% <0.00%> (-7.35%) ⬇️
scripts/evm_stress/main.go 0.00% <0.00%> (ø)
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@codchen codchen force-pushed the tony/scheduler-conflict-log branch from ee0266f to bbf1182 Compare May 5, 2026 04:29
@codchen codchen changed the title add OCC scheduler key conflict logs and EndBlock latency logs Optimize conflict-heavy EVM transfer block execution May 6, 2026
codchen and others added 13 commits May 6, 2026 11:47
- Track per-key conflict counts in OCC scheduler (both sei-cosmos and
  giga variants); emit a single aggregated log line per block when
  conflicts occur, showing which store keys were hot
- Add ValidateTransactionStateWithKeys to MultiVersionStore interface so
  conflicting key strings are surfaced alongside conflict indices without
  breaking existing ValidateTransactionState callers
- Add per-section latency logging to x/evm/keeper EndBlock covering
  tx-hash removal, receipt migration, storage pruning, base fee
  adjustment, coinbase resolution, deferred info fetch, deferred info
  processing, surplus distribution, and bloom computation

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Log per-block execution-only time (ProcessProposal + FinalizeBlock + Commit)
in BaseApp as "execution block time" to complement the block-time monitor in
the stress test. Also add the evm_stress.sh script and Go load tester that
spins up a local seid node and hammers it with 500 TPS of EVM transfers from
rotating fresh accounts.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Drop the per-section EndBlock latency instrumentation from
x/evm/keeper/abci.go. For the OCC scheduler key conflict log, encode
each key as hex before printing so raw store bytes are readable, except
for "globalAccountNumber" which is kept as a plain string.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Drop the block-time and TPS console output from the stress tester since
those numbers now come from the node logs. Add block_txs to the
"execution block time" log line in BaseApp so each block's tx count is
visible alongside the timing breakdown.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Always abort optimistic processing immediately so blocks are only ever
executed in FinalizeBlock. The completion signal is still sent so
FinalizeBlocker does not stall on <-completion. This eliminates the
duplicate OCC and execution-block-time log lines that came from the
same block being executed twice.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Restore the optimistic ProcessBlock goroutine in ProcessProposalHandler
(reverts "disable optimistic execution in ProcessProposal").

Fix duplicate log lines in evm_stress.sh: tail -F re-opens the log file
by name when it is truncated by a new run, causing stale tail processes
from prior runs to re-emit every line. Add a pre-flight pkill to kill
lingering watchers, and switch to process-group kill in cleanup so tail
and grep children are reliably terminated.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
In findConflicts, capture the StoreKey when iterating multiVersionStores
and prefix each conflict key as "storeName/rawKey" before accumulating
into conflictKeyCounts. In the ProcessAll encoding loop, split on the
first "/" to separate the store name from the raw key, then hex-encode
only the raw key (keeping "globalAccountNumber" as plain text).

Applied to both sei-cosmos/tasks and giga/deps/tasks.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace the live-funding pipeline with a genesis-based approach:
- Generate 50k deterministic sender accounts and bulk-patch genesis.json
  via an inline Python script (same technique as populate_genesis_accounts.py)
  before starting seid — no funding transactions sent to the chain.
- Each worker pops a fresh key from a pre-filled closed channel and sends
  exactly one tx (nonce=0), guaranteeing every tx has a distinct sender.
- Rate control via a shared time.Ticker at targetTPS Hz across all workers.
- Add -dump-sei-addrs flag to main.go for bech32 address generation (uses
  sei-cosmos/types/bech32 directly, no subprocess calls).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@codchen codchen force-pushed the tony/scheduler-conflict-log branch from 8d84746 to 9a4abc1 Compare May 6, 2026 03:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants