basic_process::async_wait blocks thread sporadically

[AFriendlyGuy]

I sporadically observe that basic_process::async_wait eventually enters

::waitpid(pid_, &exit_code, 0)

in process_handle_fd.hpp (around line 332). At the time this call is made, the corresponding child process is still running (verified via ps and gdb). Consequently, the parent thread blocks until the child process eventually terminates.

The call stack looks something like:

io_context::poll_one()
  -> reactive_wait_op::do_complete()
  -> basic_process_handle_fd<...>::async_wait_op_
  -> wait4()/waitpid(..., 0)

From reading the implementation of async_wait_op_, my understanding is:

1. The initial operator() calls

   waitpid(pid_, &exit_code, WNOHANG)
   

2. If the process is still running, it registers

   descriptor.async_wait(wait_read, ...)
   

3. The completion handler then calls

   waitpid(pid_, &exit_code, 0)
   

This seems to assume that the descriptor becomes readable only when the child process can be reaped.

However, in the failing cases, the second operator() is entered although the child process is still running. As a consequence, the blocking waitpid(..., 0) blocks the calling thread.

Environment:

• Boost 1.91.0
• Boost.Process V2
• Linux 6.12.0
• BOOST_PROCESS_V2_PIDFD_OPEN == 1
• BOOST_PROCESS_V2_HAS_PROCESS_HANDLE == 1

The application uses Boost.Fiber and calls io_context::poll_one() from a fiber.

Do you have any idea, what the problem could be?

[klemens-morgenstern]

Here's is the relevant documentation for pidfd_open:

 When the task that it refers to terminates and becomes a
          zombie, these interfaces indicate the file descriptor as
          readable (EPOLLIN).  When the task is reaped, these interfaces
          produce a hangup event (EPOLLHUP).

The reason the second call blocks is so we're reaping the zombie. What kind of time frame are we looking at? Is it a few seconds in an environment where you create lots of processes?

I think I could make the reaping of the zombie process asynchronous too,
but I that will only help you if that's actually what's blocking.

[AFriendlyGuy]

In my case the child process is not a zombie, it is normally running: I can step in with gdb. When the child process is done after several minutes or more, the parent process continues normally.

There were around 20 processes started immediately after each other, when I observed the behaviour once. In that particually case, the child process started lasted was the process, which caused the blocking.

[klemens-morgenstern]

1. Are using the process api to create the process or are you supplying your own process fd?
2. Are your child processes creating further children?
3. Are you using process groups?

[AFriendlyGuy]

I use the process api to create the processes (see below). The child processes do not create further children, only threads. I do not use process groups. Here is the relevant part of the code I am using for generating processes:

std::map<SomeId, boost::process> processMap;
while(running) {
  Task task = taskChannel.value_pop();
  // generate input file and someId
  boost::process::process newProcess{ioContext, "fooBar", { inputFileName}};
  boost::process::process& actualProcess = processMap.insert(std::make_pair(someId, std::move(newProcess))).first->second;
  actualProcess.async_wait([this](std::error_code, const int exitCode) { onProcessFinished(exitCode);  });      
}

taskChannel is of type boost::fibers::buffered_channel.