From: Kai-Hsun Chen Date: Thu, 18 Dec 2025 22:54:59 +0000 (-0800) Subject: [python] aio: fix race condition causing `asyncio.run()` to hang forever during the... X-Git-Url: https://git.feebdaed.xyz/?a=commitdiff_plain;h=ddbfe03ab7bab5d64ea709b11895bc49bca488e7;p=0xmirror%2Fgrpc.git [python] aio: fix race condition causing `asyncio.run()` to hang forever during the shutdown process (#40989) # Root cause * gRPC AIO creates a Unix domain socket pair, and the current thread passes the read socket to the event loop for reading, while the write socket is passed to a thread for polling events and writing a byte into the socket. * However, during the shutdown process, the event loop stops reading the read socket without closing it before the polling thread receives the final event to exit the thread. * The shutdown process will hang if (1) the event loop stops reading the read socket before the polling thread receives the final event to exit the thread, and (2) the polling process stuck at `write` syscall. * The `write` syscall may get stuck at [sock_alloc_send_pskb](https://elixir.bootlin.com/linux/v5.15/source/net/core/sock.c#L2463) when there is not enough socket buffer space for the write socket. Hence, the polling thread hangs at write and cannot continue to the next iteration to retrieve the final event. As a result, the event loop no longer reads the read socket, so the allocable buffer size for the write socket does not increase any longer. Therefore, the current thread hangs when waiting for the polling thread to `join()`. * `asyncio` will shutdown the default executor (`ThreadPoolExecutor`) when `asyncio.run(...)` finishes. Hence, it hangs because some threads can't join. # Reproduction * Step 0: Reduce the socket buffer size to increase the probability to reproduce the issue. ```sh sysctl -w net.core.rmem_default=8192 sysctl -w net.core.rmem_default=8192 ``` * Step 1: Manually update `unistd.write(fd, b'1', 1)` to `unistd.write(fd, b'1' * 4096, 4096)`. The goal is to make write (4096 bytes per write) faster than read (1 byte per read), thereby filling the write buffer nearly full. https://github.com/grpc/grpc/blob/8e67cb088d3709ae74c1ff31d1655bea6c2b86c0/src/python/grpcio/grpc/_cython/_cygrpc/aio/completion_queue.pyx.pxi#L31 * Step 2: Create an `aio.insecure_channel` and use it to send 100 requests with at most 10 in-flight requests. After all requests finish, the shutdown process will be triggered, and it's highly likely to hang if you follow Steps 0 and 1 correctly. In my case, my reproduction script reproduces the issue 10 out of 10 times. * Step 3: If it hangs, check the following information: * `ss -xpnm state connected | grep $PID` => You will find there are two sockets that belong to the same socket pair, and one has non-zero bytes in the read buffer while the other has non-zero bytes in the write buffer. In addition, write buffer should be close to `net.core.rmem_default`. * Check the stack of the `_poller_thread` by running `cat /proc/$PID/task/$TID/stack`. The thread is stuck at `sock_alloc_send_pskb` because there is not enough buffer space to finish the `write` syscall. * Use GDB to find the `_poller_thread` and make sure it's stuck at `write()`, then print its `$rdi` to confirm that the FD is the one with a non-zero write buffer in the socket. # Test Follow Steps 0, 1, and 2 in the 'Reproduction' section with this PR. It doesn't hang in 10 out of 10 cases. Closes #40989 COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/40989 from kevin85421:asyncio-hang ff74508a2c29e7c71dfe88365d1178f901d69787 PiperOrigin-RevId: 846425459 --- diff --git a/src/python/grpcio/grpc/_cython/_cygrpc/aio/completion_queue.pyx.pxi b/src/python/grpcio/grpc/_cython/_cygrpc/aio/completion_queue.pyx.pxi index 8700ce1b6b..6b5e0b2a76 100644 --- a/src/python/grpcio/grpc/_cython/_cygrpc/aio/completion_queue.pyx.pxi +++ b/src/python/grpcio/grpc/_cython/_cygrpc/aio/completion_queue.pyx.pxi @@ -131,14 +131,20 @@ cdef class PollerCompletionQueue(BaseCompletionQueue): for loop in self._loops: self._loops.get(loop).close() + # Close the read socket to prevent the `_poller_thread` from blocking on a `write` syscall + # when the Unix-domain socket buffer is full. Once the loops above are closed, the read + # socket is no longer being read, so close it to avoid `write` syscall hangs. + # + # See `sock_alloc_send_pskb` for more details about these `write` syscall hangs. + self._read_socket.close() + # TODO(https://github.com/grpc/grpc/issues/22365) perform graceful shutdown grpc_completion_queue_shutdown(self._cq) while not self._shutdown: self._poller_thread.join(timeout=_POLL_AWAKE_INTERVAL_S) grpc_completion_queue_destroy(self._cq) - # Clean up socket resources - self._read_socket.close() + # Clean up the write socket self._write_socket.close() def _handle_events(self, object context_loop):