fix(core/txpool): fix Sync deadlock in pool loop #28837 by gzliudan · Pull Request #2542 · XinFinOrg/XDPoSChain

gzliudan · 2026-08-26T00:14:56Z

Summary

Fixes a pool.Sync() deadlock that surfaced as a 45-minute test timeout on core/txpool/locals.

Symptoms

CI hung until the 45-minute test timeout, with running tests: TestTrackAllSameNonceReplacement. The goroutine dump showed the test goroutine blocked at TxPool.Sync()'s <-waiter while TxPool.loop idled in select with no reset goroutine running — Sync() had sent its sync request and the loop had received it, but the waiter was never signaled back.

Ref: https://github.com/XinFinOrg/XDPoSChain/actions/runs/32908385602/job/97997298962

ok  	github.com/XinFinOrg/XDPoSChain/core/txpool/legacypool	7.075s	coverage: 43.0% of statements
coverage: 74.1% of statements
panic: test timed out after 45m0s
	running tests:
		TestTrackAllSameNonceReplacement (45m0s)

goroutine 235 [running]:
testing.(*M).startAlarm.func1()
	/opt/hostedtoolcache/go/1.25.13/x64/src/testing/testing.go:2682 +0x359
created by time.goFunc
	/opt/hostedtoolcache/go/1.25.13/x64/src/time/sleep.go:215 +0x2d

goroutine 1 [chan receive, 44 minutes]:
testing.tRunner.func1()
	/opt/hostedtoolcache/go/1.25.13/x64/src/testing/testing.go:1891 +0x451
testing.tRunner(0xc0000a4540, 0xc00029fc70)
	/opt/hostedtoolcache/go/1.25.13/x64/src/testing/testing.go:1940 +0x123
testing.runTests(0xc0000124b0, {0x11646e0, 0xa, 0xa}, {0x11c3040?, 0x87da55bf9198f5db?, 0x11c0520?})
	/opt/hostedtoolcache/go/1.25.13/x64/src/testing/testing.go:2475 +0x4b4
testing.(*M).Run(0xc0001b2e60)
	/opt/hostedtoolcache/go/1.25.13/x64/src/testing/testing.go:2337 +0x63a
main.main()
	_testmain.go:73 +0x9b

goroutine 39 [chan receive, 44 minutes]:
github.com/XinFinOrg/XDPoSChain/core/txpool.(*TxPool).Sync(...)
	/home/runner/work/XDPoSChain/XDPoSChain/src/XDPoSChain/core/txpool/txpool.go:511
github.com/XinFinOrg/XDPoSChain/core/txpool/locals.newTestEnv(0xc000103340, 0xa, 0x0, {0x0, 0x0})
	/home/runner/work/XDPoSChain/XDPoSChain/src/XDPoSChain/core/txpool/locals/tx_tracker_test.go:90 +0x490
github.com/XinFinOrg/XDPoSChain/core/txpool/locals.TestTrackAllSameNonceReplacement(0xc000103340)
	/home/runner/work/XDPoSChain/XDPoSChain/src/XDPoSChain/core/txpool/locals/tx_tracker_test.go:203 +0x50
testing.tRunner(0xc000103340, 0xc022b8)
	/opt/hostedtoolcache/go/1.25.13/x64/src/testing/testing.go:1934 +0xea
created by testing.(*T).Run in goroutine 1
	/opt/hostedtoolcache/go/1.25.13/x64/src/testing/testing.go:1997 +0x465

goroutine 13 [chan receive, 44 minutes]:
github.com/XinFinOrg/XDPoSChain/core.(*txSenderCacher).cache(0x0?)
	/home/runner/work/XDPoSChain/XDPoSChain/src/XDPoSChain/core/sender_cacher.go:72 +0x35
created by github.com/XinFinOrg/XDPoSChain/core.newTxSenderCacher in goroutine 8
	/home/runner/work/XDPoSChain/XDPoSChain/src/XDPoSChain/core/sender_cacher.go:64 +0x74

goroutine 14 [chan receive, 44 minutes]:
github.com/XinFinOrg/XDPoSChain/core.(*txSenderCacher).cache(0x0?)
	/home/runner/work/XDPoSChain/XDPoSChain/src/XDPoSChain/core/sender_cacher.go:72 +0x35
created by github.com/XinFinOrg/XDPoSChain/core.newTxSenderCacher in goroutine 8
	/home/runner/work/XDPoSChain/XDPoSChain/src/XDPoSChain/core/sender_cacher.go:64 +0x74

goroutine 15 [chan receive, 44 minutes]:
github.com/XinFinOrg/XDPoSChain/core.(*txSenderCacher).cache(0x0?)
	/home/runner/work/XDPoSChain/XDPoSChain/src/XDPoSChain/core/sender_cacher.go:72 +0x35
created by github.com/XinFinOrg/XDPoSChain/core.newTxSenderCacher in goroutine 8
	/home/runner/work/XDPoSChain/XDPoSChain/src/XDPoSChain/core/sender_cacher.go:64 +0x74

goroutine 16 [chan receive, 44 minutes]:
github.com/XinFinOrg/XDPoSChain/core.(*txSenderCacher).cache(0x0?)
	/home/runner/work/XDPoSChain/XDPoSChain/src/XDPoSChain/core/sender_cacher.go:72 +0x35
created by github.com/XinFinOrg/XDPoSChain/core.newTxSenderCacher in goroutine 8
	/home/runner/work/XDPoSChain/XDPoSChain/src/XDPoSChain/core/sender_cacher.go:64 +0x74

goroutine 156 [select]:
github.com/XinFinOrg/XDPoSChain/core.(*BlockChain).futureBlocksLoop(0xc000004808)
	/home/runner/work/XDPoSChain/XDPoSChain/src/XDPoSChain/core/blockchain.go:2669 +0xb4
sync.(*WaitGroup).Go.func1()
	/opt/hostedtoolcache/go/1.25.13/x64/src/sync/waitgroup.go:239 +0x4a
created by sync.(*WaitGroup).Go in goroutine 39
	/opt/hostedtoolcache/go/1.25.13/x64/src/sync/waitgroup.go:237 +0x73

goroutine 157 [select, 44 minutes]:
github.com/XinFinOrg/XDPoSChain/core/txpool/legacypool.(*LegacyPool).scheduleReorgLoop(0xc0003e6e00)
	/home/runner/work/XDPoSChain/XDPoSChain/src/XDPoSChain/core/txpool/legacypool/legacypool.go:1257 +0x28c
sync.(*WaitGroup).Go.func1()
	/opt/hostedtoolcache/go/1.25.13/x64/src/sync/waitgroup.go:239 +0x4a
created by sync.(*WaitGroup).Go in goroutine 39
	/opt/hostedtoolcache/go/1.25.13/x64/src/sync/waitgroup.go:237 +0x73

goroutine 158 [select]:
github.com/XinFinOrg/XDPoSChain/core/txpool/legacypool.(*LegacyPool).loop(0xc0003e6e00)
	/home/runner/work/XDPoSChain/XDPoSChain/src/XDPoSChain/core/txpool/legacypool/legacypool.go:364 +0x15b
sync.(*WaitGroup).Go.func1()
	/opt/hostedtoolcache/go/1.25.13/x64/src/sync/waitgroup.go:239 +0x4a
created by sync.(*WaitGroup).Go in goroutine 39
	/opt/hostedtoolcache/go/1.25.13/x64/src/sync/waitgroup.go:237 +0x73

goroutine 159 [select, 44 minutes]:
github.com/XinFinOrg/XDPoSChain/core/txpool.(*TxPool).loop(0xc0004303f0, 0xc000409688)
	/home/runner/work/XDPoSChain/XDPoSChain/src/XDPoSChain/core/txpool/txpool.go:234 +0x4ec
created by github.com/XinFinOrg/XDPoSChain/core/txpool.New in goroutine 39
	/home/runner/work/XDPoSChain/XDPoSChain/src/XDPoSChain/core/txpool/txpool.go:124 +0x37f
FAIL	github.com/XinFinOrg/XDPoSChain/core/txpool/locals	2700.008s
ok  	github.com/XinFinOrg/XDPoSChain/core/types	0.625s	coverage: 43.3% of statements
ok  	github.com/XinFinOrg/XDPoSChain/core/vm	0.117s	coverage: 46.9% of statements
ok  	github.com/XinFinOrg/XDPoSChain/core/vm/privacy	6.150s	coverage: 82.2% of statements
ok  	github.com/XinFinOrg/XDPoSChain/core/vm/program	0.004s	coverage: 76.0% of statements
ok  	github.com/XinFinOrg/XDPoSChain/core/vm/runtime	0.055s	coverage: 90.2% of statements
FAIL
util.go:48: exit status 1
exit status 1
Error: Process completed with exit code 1.

Root cause

Upstream geth notifies the sync waiter in the resetDone branch of TxPool.loop with a blocking send:

if resetWaiter != nil && !resetForced {
    resetWaiter <- nil
    resetWaiter = nil
}

This fork's commit a9be217e6 (#2132) changed it to a non-blocking send followed by an unconditional clear:

if resetWaiter != nil && !resetForced {
    select {
    case resetWaiter <- nil:
    default:
    }
    resetWaiter = nil
}

Deadlock chain:

Sync() sends its waiter over p.sync; the loop receives it and sets resetWaiter/resetForced=true;
Sync() then runs return <-waiter — a scheduling gap exists between these two steps;
if the reset finishes before Sync() reaches <-waiter, resetWaiter <- nil takes the default branch (no receiver yet), the notification is dropped, and resetWaiter is unconditionally cleared;
Sync()'s waiter never receives a value → blocked forever.

Why only CI hit it: coverage mode inserts counters at every statement, widening the descheduling window between sending the waiter and starting to receive; parallel high load worsens it. Plain local runs almost never trigger it.

Fix

Restore the blocking send used upstream, which cannot lose the notification, and apply the same to the loop-termination notification so a pool closed while Sync() is pending also signals the waiter. Add TestSyncCompletes, which exercises repeated Sync() calls and hangs under -cover if the non-blocking notification regresses.

Verification

core/txpool/locals: 100 consecutive runs under -cover all pass (previously timed out at the same parameters);
core/txpool: full package passes; TestSyncCompletes passes 100 runs under -cover and -race;
regression check: with the fix reverted (non-blocking send), TestSyncCompletes hangs and times out under -cover, confirming it catches the deadlock.

Compatibility

Internal txpool concurrency control only; no protocol, RPC, or state-transition changes.

coderabbitai · 2026-08-26T00:15:05Z

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 08160e0e-a4d8-4df6-9d59-bcc948efb804

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

Copilot

Pull request overview

Fixes txpool reset synchronization races by aligning head-event subscription and waiter signaling with upstream geth.

Changes:

Subscribes to chain-head events synchronously during pool creation.
Guarantees delivery of reset completion notifications.
Adds regression tests for synchronization and head-event handling.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File	Description
`core/txpool/txpool.go`	Updates subscription lifecycle and reset waiter signaling.
`core/txpool/txpool_sync_test.go`	Adds synchronization and head-event regression tests.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

core/txpool/txpool_sync_test.go:91

Calling Sync synchronously means a regression recreates the original behavior: this test stalls the entire package until the global CI timeout (45 minutes in the referenced run) instead of reporting a prompt failure. Run the call in a goroutine and select on its result with a bounded timeout so the deadlock test fails quickly and diagnostically.

	for i := 0; i < 100; i++ {
		if err := pool.Sync(); err != nil {

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

…tion ethereum#28837 Sync() could block forever: the resetDone branch notified the sync waiter with a non-blocking send and then unconditionally cleared it. When Sync() had not yet reached its receive (a scheduling gap after it sent its waiter), the notification was dropped, leaving the waiter never signaled. Restore the blocking send used upstream geth, which cannot lose the notification, and do the same for the loop-termination notification so a pool closed while Sync() is pending also signals the waiter. Add TestSyncCompletes which exercises repeated Sync() calls and would hang under -cover if the non-blocking notification regressed.

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

gzliudan force-pushed the fix-txpool-sync-head-events branch 3 times, most recently from 5c58063 to 25aaa21 Compare August 26, 2026 00:41

gzliudan requested review from AnilChinchawale, anunay-xin, benjamin202410, liam-lai and wanwiset25 and a balanced review from Copilot August 26, 2026 00:48

Copilot started reviewing on behalf of gzliudan August 26, 2026 00:49 View session

Copilot AI reviewed Aug 26, 2026

View reviewed changes

Comment thread core/txpool/txpool.go Outdated

Comment thread core/txpool/txpool_sync_test.go Outdated

Comment thread core/txpool/txpool_sync_test.go Outdated

gzliudan force-pushed the fix-txpool-sync-head-events branch from 25aaa21 to e6b1581 Compare August 26, 2026 01:06

gzliudan changed the title ~~fix(core/txpool): fix Sync deadlock and lost head events in pool loop~~ fix(core/txpool): fix Sync deadlock by blocking reset waiter notification Aug 26, 2026

gzliudan requested a balanced review from Copilot August 26, 2026 01:19

Copilot started reviewing on behalf of gzliudan August 26, 2026 01:20 View session

Copilot AI reviewed Aug 26, 2026

View reviewed changes

Comment thread core/txpool/txpool.go

gzliudan changed the title ~~fix(core/txpool): fix Sync deadlock by blocking reset waiter notification~~ fix(core/txpool): fix Sync deadlock and lost head events in pool loop #28837 Aug 26, 2026

gzliudan force-pushed the fix-txpool-sync-head-events branch 2 times, most recently from 60eb82c to 27dc451 Compare August 26, 2026 01:49

gzliudan requested a balanced review from Copilot August 26, 2026 01:51

Copilot started reviewing on behalf of gzliudan August 26, 2026 01:51 View session

Copilot AI reviewed Aug 26, 2026

View reviewed changes

Comment thread core/txpool/txpool_sync_test.go Outdated

Comment thread core/txpool/txpool_sync_test.go Outdated

gzliudan force-pushed the fix-txpool-sync-head-events branch from 27dc451 to 34aadee Compare August 26, 2026 02:08

gzliudan requested a balanced review from Copilot August 26, 2026 02:30

Copilot started reviewing on behalf of gzliudan August 26, 2026 02:31 View session

Copilot AI reviewed Aug 26, 2026

View reviewed changes

Comment thread core/txpool/txpool.go

gzliudan changed the title ~~fix(core/txpool): fix Sync deadlock and lost head events in pool loop #28837~~ fix(core/txpool): fix Sync deadlock in pool loop #28837 Aug 26, 2026

Navigation Menu

Uh oh!

fix(core/txpool): fix Sync deadlock in pool loop #28837 - #2542

fix(core/txpool): fix Sync deadlock in pool loop #28837#2542
gzliudan wants to merge 1 commit into
XinFinOrg:dev-upgradefrom
gzliudan:fix-txpool-sync-head-events

gzliudan commented Aug 26, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Aug 26, 2026 •

edited

Loading

Review skipped

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

gzliudan commented Aug 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Symptoms

Root cause

Fix

Verification

Compatibility

Uh oh!

coderabbitai Bot commented Aug 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

gzliudan commented Aug 26, 2026 •

edited

Loading

coderabbitai Bot commented Aug 26, 2026 •

edited

Loading