Non-worker mode: thread-pool deadlock ~5s after startup under concurrent load (1.12.6, ZTS) — threads wedged in regularThread.waitForRequest / transitionToNewHandler

[cyppe]

Summary

On FrankenPHP 1.12.6 (PHP 8.4.23 ZTS, Caddy 2.11.4), in classic non-worker mode, a container intermittently deadlocks its entire PHP thread pool a few seconds after a clean startup, under the burst of concurrent traffic that a rolling deploy sends the instant the server binds. Once wedged: :8000 stops serving all requests, the Caddy admin endpoint :2019 stays fully responsive, the container never recovers (a curl :8000/up health check hangs forever), and only a restart clears it. It's probabilistic — one of two identical hosts wedges on a given deploy while the other is fine.

This is in the same area as #2250 (ThreadState.WaitFor / go_frankenphp_main_thread_is_ready), but the trigger is different: here the server binds, logs server running, and serves real traffic for ~5 s before the whole pool deadlocks — not a never-binds init hang.

Environment

• FrankenPHP v1.12.6, Caddy v2.11.4, PHP 8.4.23 (ZTS) — image dunglas/frankenphp:1.12.6-php8.4-bookworm (amd64)
• Classic / regular mode (php_server; no worker directive)
• frankenphp { num_threads 300 max_threads 300 max_wait_time 10s } — fixed thread count. Pinning num_threads == max_threads to disable dynamic scaling did not prevent it.
• Docker, 32-core / ~60 GB host, non-privileged, listening on :8000; a reverse proxy cuts live traffic over to the new container immediately on deploy.
• Also reproduced on 1.12.4; upgrading 1.12.4 → 1.12.6 did not fix it.

Timeline (from the container's own logs)

"FrankenPHP started 🐘"   php_version=8.4.23  num_threads=300  max_threads=300  max_requests=0
"server running"          name=srv0
... ~5 seconds of normal request handling (HTTP 200s) ...
<no request handled after ~T+5s; :8000 wedged; :2019 still answering; never recovers>

Goroutine profile at the deadlock (462 goroutines)

Two dominant clusters (?debug=1):

~151 PHP threads stuck in waitForRequest, reached via a cgo callback (locked to thread):

151 @ ...
#  github.com/dunglas/frankenphp.(*regularThread).waitForRequest+0x346          /go/src/app/threadregular.go:110
#  github.com/dunglas/frankenphp.(*regularThread).beforeScriptExecution+0x584   /go/src/app/threadregular.go:53
#  github.com/dunglas/frankenphp.(*phpThread).transitionToNewHandler+0x66       /go/src/app/phpthread.go:177
#  github.com/dunglas/frankenphp.(*inactiveThread).beforeScriptExecution+0x10e  /go/src/app/threadinactive.go:26
#  github.com/dunglas/frankenphp.(*inactiveThread).beforeScriptExecution+0xfe   /go/src/app/threadinactive.go:36
#  github.com/dunglas/frankenphp.go_frankenphp_before_script_execution+0x16f    /go/src/app/phpthread.go:237
#  _cgoexp_..._go_frankenphp_before_script_execution+0x1a                       /go/src/app/phpmainthread.go:280
#  runtime.cgocallbackg1 / runtime.cgocallbackg / runtime.cgocallback

~148 incoming requests parked waiting for a PHP thread:

148 @ ...
#  github.com/dunglas/frankenphp.handleRequestWithRegularPHPThreads+0x828   /go/src/app/threadregular.go:148
#  github.com/dunglas/frankenphp.ServeHTTP+0x710                            /go/src/app/frankenphp.go:427
#  ...caddyhttp request pipeline...

Representative full stacks (?debug=2):

goroutine 17 [chan receive, 8 minutes, locked to thread]:
github.com/dunglas/frankenphp/internal/state.(*ThreadState).WaitFor(0x..., {0x..., 0x2, 0x2})
	/go/src/app/internal/state/state.go:167 +0x158
github.com/dunglas/frankenphp.go_frankenphp_main_thread_is_ready()
	/go/src/app/phpmainthread.go:238 +0x7d

goroutine 98 [select, 8 minutes, locked to thread]:
github.com/dunglas/frankenphp.(*regularThread).waitForRequest(0x...)
	/go/src/app/threadregular.go:110 +0x347
github.com/dunglas/frankenphp.(*regularThread).beforeScriptExecution(0x...)
	/go/src/app/threadregular.go:53 +0x585
github.com/dunglas/frankenphp.(*phpThread).transitionToNewHandler(0x...)
	/go/src/app/phpthread.go:177 +0x67
github.com/dunglas/frankenphp.(*inactiveThread).beforeScriptExecution(0x...)
	/go/src/app/threadinactive.go:26 +0x10f
github.com/dunglas/frankenphp.(*inactiveThread).beforeScriptExecution(0x...)
	/go/src/app/threadinactive.go:36 +0xff
github.com/dunglas/frankenphp.go_frankenphp_before_script_execution(0x0?)
	/go/src/app/phpthread.go:237 +0x170

Interpretation (tentative)

Looks like a thread-pool state-machine deadlock: ~half the PHP threads are wedged mid-transitionToNewHandler → waitForRequest (inside a cgo callback, locked to thread), so they never pick up the ~148 requests parked in handleRequestWithRegularPHPThreads, while go_frankenphp_main_thread_is_ready is still blocked in ThreadState.WaitFor. The trigger appears to be the burst of concurrent requests arriving in the first seconds after server running, while threads are still transitioning inactive → regular.

Notes

• Mutex profiling isn't enabled in this build (/debug/pprof/mutex sampling period = 0), so I couldn't capture the lock holder. Happy to enable SetMutexProfileFraction and re-capture, or share the full 462-goroutine ?debug=2 dump (~14k lines).
• Reproduction is probabilistic on deploy under live traffic. I can try to build a minimal repro (bind, then immediately hit with N concurrent requests during thread warm-up) if useful.
• Possibly related: FrankenPHP v1.12.0: HTTP server never starts - hangs in initPHPThreads/WaitFor (non-worker mode) #2250 (same WaitFor/main_thread_is_ready code), Permanent worker starvation after maximum execution time worker restarts #2205 (worker starvation after restarts).