Skip to content

Guard the shutdown log wakeup against an unallocated notify array - #13541

Draft
bryancall wants to merge 3 commits into
apache:masterfrom
bryancall:fix/guard-preproc-notify-on-shutdown
Draft

Guard the shutdown log wakeup against an unallocated notify array#13541
bryancall wants to merge 3 commits into
apache:masterfrom
bryancall:fix/guard-preproc-notify-on-shutdown

Conversation

@bryancall

@bryancall bryancall commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Log::preproc_notify is allocated in Log::create_threads(), which runs only once logging is fully enabled. Until then the pointer is null, while Log::preproc_threads has already been set to a nonzero value by Log::init(). AutoStopCont::mainEvent walked the array anyway:

// Wake preproc threads to drain remaining log buffers before exit.
for (int i = 0; i < Log::preproc_threads; i++) {
  Log::preproc_notify[i].signal();
}

The result is a SIGSEGV at address zero.

Two distinct ways to get there, which is why the guard is phrased around the array rather than around a race:

  • Startup window. Log::init() runs at traffic_server.cc:2382; Log::load_config() does not run until :2407. In between sit api_init() and plugin_yaml_init(), so the window spans every plugin's TSPluginInit. SignalContinuation is already scheduled by then, and proxy.config.stop.shutdown_timeout defaults to 0, so a SIGTERM during a slow plugin init lands straight in AutoStopCont.
  • Permanently. Log::load_config() returns without calling init_when_enabled() when config_flags & LOGCAT is set, so for the log-only tools the array is null for the life of the process.

#13472 added exactly this guard to the two call sites in LogObject.cc and described the same failure, but did not cover this third copy in traffic_server.cc, which was introduced separately by #13065.

Nothing is lost by skipping the signal. preproc_notify is assigned in exactly one place and never reset, and the preproc threads are spawned in that same function immediately after the allocation, so a null array means no preproc thread exists and the loop could not have drained anything.

Scope

The fourth use, in PeriodicWakeup::wakeup, is deliberately left alone: it is scheduled on the line immediately after create_threads() returns, so the array is always allocated by then. This PR closes the last exposed site.

Worth noting as possible follow-up rather than part of this change: allocating preproc_notify in Log::init() once preproc_threads is final would establish the invariant and let all three guards go away.

Impact

Startup window only. start_HttpProxyServer() runs well after Log::load_config(), so a process that reaches this path never served traffic. The cost is a crash and a core file where a clean exit belonged, not a traffic impact.

Testing

Built with the dev-asan preset (-fsanitize=address,undefined) on Fedora 44 / gcc 16, -Werror clean.

  • unit (ctest): 174/174 passed
  • in-binary regression (traffic_server -R 1): 67 passed, REGRESSION_TEST DONE: PASSED
  • autest: 476 passed / 65 failed / 40 skipped, over 561 tests (remap_acl and remap_acl_yaml excluded for runtime)

The 65 autest failures are pre-existing environment failures on this box, not regressions. I built unmodified master at the same base commit (8aebe2c706) and ran the identical 561-test selection: it produces the same 476/65/40 and the same 61 distinct failing tests, so the failing set matches the baseline exactly. They are concentrated in TLS, QUIC/HTTP3, and config-reload tests.

Log::init() sets Log::preproc_threads to 1 immediately, but
Log::preproc_notify is not allocated until Log::create_threads().
A shutdown signal arriving between those two points reaches
AutoStopCont::mainEvent, which walks preproc_threads entries of a
null array and crashes with a SIGSEGV at address zero.

A previous change added this same guard to the two call sites in
LogObject.cc but did not cover the copy in traffic_server.cc, which
was introduced separately. Production cores show the unguarded site
still firing on builds that already carry the LogObject.cc guards.
The array is also null for the whole life of the log-only tools, where
Log::load_config returns without ever creating the log threads. Saying
only that a shutdown can arrive mid startup invites a later reader to
prove that race unreachable and drop a guard that is still load bearing.
@bryancall bryancall added this to the 11.0.0 milestone Aug 14, 2026
@bryancall bryancall self-assigned this Aug 14, 2026
@bryancall
bryancall requested a lite review from Copilot August 15, 2026 05:39

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR prevents a shutdown-time null dereference by guarding the “wake preproc threads” loop in AutoStopCont::mainEvent when Log::preproc_threads is non-zero but the Log::preproc_notify array has not been allocated yet (e.g., during the early startup window before Log::create_threads() runs).

Changes:

  • Add a nullptr guard around signaling Log::preproc_notify[i] during shutdown.
  • Expand the inline comment to document why the notify array can be unallocated at that point.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread src/traffic_server/traffic_server.cc Outdated
AutoStopCont is defined in traffic_server.cc and nothing else links it,
so traffic_logcat and traffic_logstats never reach this loop and citing
them here misleads. Command mode is the case that actually applies:
cmd_verify calls Log::init and never Log::load_config, so the array
stays null for the life of that process.
@bryancall

Copy link
Copy Markdown
Contributor Author

Adding the production evidence behind this, since the description referred to crash reports without showing one.

Stack as reported, on a 10.0.x-based build:

EventNotify::signal()
(anonymous namespace)::AutoStopCont::mainEvent(int, Event*)
handleEvent
EThread::process_event(Event*, int, long)
EThread::execute_regular()
execute
EThread::execute()
spawn_thread_internal

Two occurrences in a seven day window, on two separate machines.

The attribution is unambiguous: Log::preproc_notify[i].signal() is the only EventNotify::signal() call anywhere in AutoStopCont::mainEvent, so a fault at that frame from that caller can only be this loop. That is the line this PR guards.

Two caveats so this is not read as more than it is. The faulting address was not captured for these two events, so I am not claiming a confirmed null dereference from this data alone. What the data establishes is that the loop faults in production, and inspection establishes that the array can legitimately be null there. #13472 described the identical failure mode at the two LogObject.cc call sites and fixed those.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants