Guard the shutdown log wakeup against an unallocated notify array - #13541
Guard the shutdown log wakeup against an unallocated notify array#13541bryancall wants to merge 3 commits into
Conversation
Log::init() sets Log::preproc_threads to 1 immediately, but Log::preproc_notify is not allocated until Log::create_threads(). A shutdown signal arriving between those two points reaches AutoStopCont::mainEvent, which walks preproc_threads entries of a null array and crashes with a SIGSEGV at address zero. A previous change added this same guard to the two call sites in LogObject.cc but did not cover the copy in traffic_server.cc, which was introduced separately. Production cores show the unguarded site still firing on builds that already carry the LogObject.cc guards.
The array is also null for the whole life of the log-only tools, where Log::load_config returns without ever creating the log threads. Saying only that a shutdown can arrive mid startup invites a later reader to prove that race unreachable and drop a guard that is still load bearing.
There was a problem hiding this comment.
Pull request overview
This PR prevents a shutdown-time null dereference by guarding the “wake preproc threads” loop in AutoStopCont::mainEvent when Log::preproc_threads is non-zero but the Log::preproc_notify array has not been allocated yet (e.g., during the early startup window before Log::create_threads() runs).
Changes:
- Add a
nullptrguard around signalingLog::preproc_notify[i]during shutdown. - Expand the inline comment to document why the notify array can be unallocated at that point.
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
AutoStopCont is defined in traffic_server.cc and nothing else links it, so traffic_logcat and traffic_logstats never reach this loop and citing them here misleads. Command mode is the case that actually applies: cmd_verify calls Log::init and never Log::load_config, so the array stays null for the life of that process.
|
Adding the production evidence behind this, since the description referred to crash reports without showing one. Stack as reported, on a 10.0.x-based build: Two occurrences in a seven day window, on two separate machines. The attribution is unambiguous: Two caveats so this is not read as more than it is. The faulting address was not captured for these two events, so I am not claiming a confirmed null dereference from this data alone. What the data establishes is that the loop faults in production, and inspection establishes that the array can legitimately be null there. #13472 described the identical failure mode at the two |
Log::preproc_notifyis allocated inLog::create_threads(), which runs only once logging is fully enabled. Until then the pointer is null, whileLog::preproc_threadshas already been set to a nonzero value byLog::init().AutoStopCont::mainEventwalked the array anyway:The result is a SIGSEGV at address zero.
Two distinct ways to get there, which is why the guard is phrased around the array rather than around a race:
Log::init()runs attraffic_server.cc:2382;Log::load_config()does not run until:2407. In between sitapi_init()andplugin_yaml_init(), so the window spans every plugin'sTSPluginInit.SignalContinuationis already scheduled by then, andproxy.config.stop.shutdown_timeoutdefaults to 0, so a SIGTERM during a slow plugin init lands straight inAutoStopCont.Log::load_config()returns without callinginit_when_enabled()whenconfig_flags & LOGCATis set, so for the log-only tools the array is null for the life of the process.#13472 added exactly this guard to the two call sites in
LogObject.ccand described the same failure, but did not cover this third copy intraffic_server.cc, which was introduced separately by #13065.Nothing is lost by skipping the signal.
preproc_notifyis assigned in exactly one place and never reset, and the preproc threads are spawned in that same function immediately after the allocation, so a null array means no preproc thread exists and the loop could not have drained anything.Scope
The fourth use, in
PeriodicWakeup::wakeup, is deliberately left alone: it is scheduled on the line immediately aftercreate_threads()returns, so the array is always allocated by then. This PR closes the last exposed site.Worth noting as possible follow-up rather than part of this change: allocating
preproc_notifyinLog::init()oncepreproc_threadsis final would establish the invariant and let all three guards go away.Impact
Startup window only.
start_HttpProxyServer()runs well afterLog::load_config(), so a process that reaches this path never served traffic. The cost is a crash and a core file where a clean exit belonged, not a traffic impact.Testing
Built with the
dev-asanpreset (-fsanitize=address,undefined) on Fedora 44 / gcc 16,-Werrorclean.traffic_server -R 1): 67 passed,REGRESSION_TEST DONE: PASSEDremap_aclandremap_acl_yamlexcluded for runtime)The 65 autest failures are pre-existing environment failures on this box, not regressions. I built unmodified master at the same base commit (
8aebe2c706) and ran the identical 561-test selection: it produces the same 476/65/40 and the same 61 distinct failing tests, so the failing set matches the baseline exactly. They are concentrated in TLS, QUIC/HTTP3, and config-reload tests.