Description
Bug report
Bug description:
When checking asyncio subprocess .returncode
for example in a busy loop, it will never be set.
Minimal example:
import asyncio
async def main():
code = """
import time, sys
time.sleep(1)
print('proc exits')
"""
proc = await asyncio.create_subprocess_exec("python", "-c", code)
while proc.returncode is None:
pass
print("proc.returncode:", proc.returncode)
if __name__ == "__main__":
asyncio.run(main())
One would expect this code to wait 1 second, print proc exits
, followed by proc.returncode: 0
. However only proc exits
is printed, meaning that the parent process's asyncio never detected the process exiting.
This is problematic for what I would imagine is a relatively common and useful usecase (as was for me finding this bug): using a loop to get output from a child process.
import asyncio
import subprocess
async def main():
code = """
import time, sys
time.sleep(1)
print('proc exits')
"""
proc = await asyncio.create_subprocess_exec("python", "-c", code, stdout=subprocess.PIPE)
while proc.returncode is None:
print("proc said:", await proc.stdout.readline())
print("proc.returncode:", proc.returncode)
if __name__ == "__main__":
asyncio.run(main())
This will print:
proc said: b'proc exits\r\n'
proc said: b''
proc said: b''
proc said: b''
proc said: b''
proc said: b''
proc said: b''
proc said: b''
proc said: b''
...
I noticed that adding a small await asyncio.sleep(0.1)
before looping again will allow the returncode to be set, but this is not ideal. Especially considering the bug happens even when using await proc.stdout.readline()
in the loop, which hints to the user that this is done efficiently and asynchronously. Having to instead think about leaving enough CPU time to the event loop to detect the process exiting is not ideal and doesn't seem pythonic to me.
For the time being I worked around this issue for my usecase by checking proc.stdout.at_eof()
which seems to work fine.
Maybe when accessing .returncode
which is a property, it could use some CPU time to see if an exit event is pending? Not sure how it is implemented under the hood so that might not even make sense.
CPython versions tested on:
3.11
Operating systems tested on:
Windows
Metadata
Metadata
Assignees
Projects
Status
Activity
kumaraditya303 commentedon Dec 6, 2024
The minimal example isn't working because you are creating a busy loop without yielding to the event loop to process events.
The following code would make it work for you:
Output:
asvetlov commentedon Dec 6, 2024
I don't think we should care about such bug reports; busy loop is not compatible with asyncio in many aspects.
WillyJL commentedon Dec 6, 2024
I understand that the busy loop without ever yielding to the event loop would have no chance to process the events in this case, but what about the example where the loop repeatedly calls
await proc.stdout.readline()
? Shouldn't that give the chance to process events?If that's not the case, thanks for pointing out
await asyncio.sleep(0)
, i had forgotten that this was an option to yield to event loop.EDIT: perhaps it could be a good idea to note this behavior in the documentation for
.returncode
?Yield to eventloop to fill .returncode
kumaraditya303 commentedon May 16, 2025
The following code works on Python 3.12 and above:
It doesn't work on 3.11 but that is in security fixes mode so closing this as won't fix.