Seid version
name: sei
server_name: seid
version: v6.6.1
go: go1.24.2
Chain ID
sei-pacific-1 (mainnet)
Describe the bug
When debug_traceBlockByNumber processes a block containing CosmWasm transactions, the trace can panic during execution. The panic is caught by recover() in recordMetricsWithError, but the semaphore slot acquired in acquireTraceSemaphore is never released. Each panic permanently consumes one slot from the traceCallSemaphore channel. Over time these accumulate until all max_concurrent_trace_calls slots are exhausted, after which every trace call is permanently rejected with "trace request rejected due to concurrency limit: server busy". Only a process restart recovers the node.
Additionally, trace calls that hit CosmWasm transactions block on a global sync.Mutex in VMWrapper.Execute (sei-wasmd/x/wasm/keeper/vm_wrapper.go:76), holding their semaphore slot while waiting. This compounds the leak by keeping slots occupied for extended periods.
To Reproduce
- Start seid with
max_concurrent_trace_calls = 500
- Confirm baseline:
curl localhost:5050/metrics | grep 'error_class="panic"' → no results
- Send concurrent
debug_traceBlockByNumber requests against blocks containing CosmWasm transactions
- Check metrics:
curl localhost:5050/metrics | grep 'error_class="panic"' → count increments
- Each panic = one permanently leaked semaphore slot
- Over time (hours/days with organic traffic) all slots are consumed → all
debug_trace* calls rejected permanently
We reproduced the full cycle on a production archive node:
- After restart: 0 panics, 454 goroutines, traces work normally
- After stress test with CosmWasm-heavy blocks: panic count went from 0 → 1 within minutes, 9 goroutines blocked on
VMWrapper.Execute mutex
- On a node in degraded state (before restart): 24 panics accumulated, 448 goroutines stuck on
VMWrapper.Execute, 472/500 semaphore slots unavailable, all trace calls rejected
Expected behavior
The semaphore slot should be released on every code path, including panic recovery. Semaphore occupancy should return to 0 when no trace calls are in-flight.
Screenshots
N/A — evidence is from Prometheus metrics and pprof goroutine dumps (details below).
Additional context
Goroutine profile of the 448 stuck goroutines on the degraded node:
sync.(*Mutex).Lock
VMWrapper.Execute (vm_wrapper.go:76)
wasm/keeper.execute (keeper.go:425)
PermissionedKeeper.Execute (contract_keeper.go:51)
msgServer.ExecuteContract (msg_server.go:100)
...
tracers.(*API).traceBlock (api.go:685)
DebugAPI.TraceBlockByNumber (tracers.go:436)
Prometheus metrics on the degraded node showing 24 panics:
sei_chain_evmrpc_request_latency_seconds_count{endpoint="debug_traceBlockByNumber",error_class="panic",success="false"} 24
20+ Sei Pacific archive backends are affected across our fleet.
This appears related to the stack overflow panic previously reported in debug_trace* calls hitting CosmWasm blocks, the difference in v6.6.1 is that recover() catches the panic so the process survives, but the semaphore slot is still lost.
Seid version
Chain ID
sei-pacific-1 (mainnet)
Describe the bug
When
debug_traceBlockByNumberprocesses a block containing CosmWasm transactions, the trace can panic during execution. The panic is caught byrecover()inrecordMetricsWithError, but the semaphore slot acquired inacquireTraceSemaphoreis never released. Each panic permanently consumes one slot from thetraceCallSemaphorechannel. Over time these accumulate until allmax_concurrent_trace_callsslots are exhausted, after which every trace call is permanently rejected with"trace request rejected due to concurrency limit: server busy". Only a process restart recovers the node.Additionally, trace calls that hit CosmWasm transactions block on a global
sync.MutexinVMWrapper.Execute(sei-wasmd/x/wasm/keeper/vm_wrapper.go:76), holding their semaphore slot while waiting. This compounds the leak by keeping slots occupied for extended periods.To Reproduce
max_concurrent_trace_calls = 500curl localhost:5050/metrics | grep 'error_class="panic"'→ no resultsdebug_traceBlockByNumberrequests against blocks containing CosmWasm transactionscurl localhost:5050/metrics | grep 'error_class="panic"'→ count incrementsdebug_trace*calls rejected permanentlyWe reproduced the full cycle on a production archive node:
VMWrapper.ExecutemutexVMWrapper.Execute, 472/500 semaphore slots unavailable, all trace calls rejectedExpected behavior
The semaphore slot should be released on every code path, including panic recovery. Semaphore occupancy should return to 0 when no trace calls are in-flight.
Screenshots
N/A — evidence is from Prometheus metrics and pprof goroutine dumps (details below).
Additional context
Goroutine profile of the 448 stuck goroutines on the degraded node:
Prometheus metrics on the degraded node showing 24 panics:
20+ Sei Pacific archive backends are affected across our fleet.
This appears related to the stack overflow panic previously reported in
debug_trace*calls hitting CosmWasm blocks, the difference in v6.6.1 is thatrecover()catches the panic so the process survives, but the semaphore slot is still lost.