Skip to content

Asyncio subprocess .returncode never gets set when busy #127661

Not planned
@WillyJL

Description

@WillyJL

Bug report

Bug description:

When checking asyncio subprocess .returncode for example in a busy loop, it will never be set.

Minimal example:

import asyncio

async def main():
    code = """
import time, sys
time.sleep(1)
print('proc exits')
    """
    proc = await asyncio.create_subprocess_exec("python", "-c", code)
    while proc.returncode is None:
        pass
    print("proc.returncode:", proc.returncode)

if __name__ == "__main__":
    asyncio.run(main())

One would expect this code to wait 1 second, print proc exits, followed by proc.returncode: 0. However only proc exits is printed, meaning that the parent process's asyncio never detected the process exiting.

This is problematic for what I would imagine is a relatively common and useful usecase (as was for me finding this bug): using a loop to get output from a child process.

import asyncio
import subprocess

async def main():
    code = """
import time, sys
time.sleep(1)
print('proc exits')
    """
    proc = await asyncio.create_subprocess_exec("python", "-c", code, stdout=subprocess.PIPE)
    while proc.returncode is None:
        print("proc said:", await proc.stdout.readline())
    print("proc.returncode:", proc.returncode)

if __name__ == "__main__":
    asyncio.run(main())

This will print:

proc said: b'proc exits\r\n'
proc said: b''
proc said: b''
proc said: b''
proc said: b''
proc said: b''
proc said: b''
proc said: b''
proc said: b''
...

I noticed that adding a small await asyncio.sleep(0.1) before looping again will allow the returncode to be set, but this is not ideal. Especially considering the bug happens even when using await proc.stdout.readline() in the loop, which hints to the user that this is done efficiently and asynchronously. Having to instead think about leaving enough CPU time to the event loop to detect the process exiting is not ideal and doesn't seem pythonic to me.

For the time being I worked around this issue for my usecase by checking proc.stdout.at_eof() which seems to work fine.

Maybe when accessing .returncode which is a property, it could use some CPU time to see if an exit event is pending? Not sure how it is implemented under the hood so that might not even make sense.

CPython versions tested on:

3.11

Operating systems tested on:

Windows

Activity

added
type-bugAn unexpected behavior, bug, or error
on Dec 6, 2024
kumaraditya303

kumaraditya303 commented on Dec 6, 2024

@kumaraditya303
Contributor

The minimal example isn't working because you are creating a busy loop without yielding to the event loop to process events.
The following code would make it work for you:

import asyncio

async def main():
    code = """
import time, sys
time.sleep(1)
print('proc exits')
    """
    proc = await asyncio.create_subprocess_exec("python", "-c", code)
    while proc.returncode is None:
        await asyncio.sleep(0)
    print("proc.returncode:", proc.returncode)

if __name__ == "__main__":
    asyncio.run(main())

Output:

python main.py 
proc exits
proc.returncode: 0
asvetlov

asvetlov commented on Dec 6, 2024

@asvetlov
Contributor

I don't think we should care about such bug reports; busy loop is not compatible with asyncio in many aspects.

WillyJL

WillyJL commented on Dec 6, 2024

@WillyJL
Author

I understand that the busy loop without ever yielding to the event loop would have no chance to process the events in this case, but what about the example where the loop repeatedly calls await proc.stdout.readline()? Shouldn't that give the chance to process events?

If that's not the case, thanks for pointing out await asyncio.sleep(0), i had forgotten that this was an option to yield to event loop.

EDIT: perhaps it could be a good idea to note this behavior in the documentation for .returncode?

added a commit that references this issue on Dec 6, 2024
kumaraditya303

kumaraditya303 commented on May 16, 2025

@kumaraditya303
Contributor

The following code works on Python 3.12 and above:

import asyncio
import subprocess

async def main():
    code = """
import time, sys
time.sleep(1)
print('proc exits')
    """
    proc = await asyncio.create_subprocess_exec("python", "-c", code, stdout=subprocess.PIPE)
    while proc.returncode is None:
        print("proc said:", await proc.stdout.readline())
    print("proc.returncode:", proc.returncode)

if __name__ == "__main__":
    asyncio.run(main())
❯ python main.py
proc said: b'proc exits\n'
proc said: b''
proc.returncode: 0

It doesn't work on 3.11 but that is in security fixes mode so closing this as won't fix.

moved this from Todo to Done in asyncioon May 16, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    stdlibPython modules in the Lib dirtopic-asynciotype-bugAn unexpected behavior, bug, or error

    Projects

    Status

    Done

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions

      Asyncio subprocess `.returncode` never gets set when busy · Issue #127661 · python/cpython

      Follow Lee on X/Twitter - Father, Husband, Serial builder creating AI, crypto, games & web tools. We are friends :) AI Will Come To Life!

      Check out: eBank.nz (Art Generator) | Netwrck.com (AI Tools) | Text-Generator.io (AI API) | BitBank.nz (Crypto AI) | ReadingTime (Kids Reading) | RewordGame | BigMultiplayerChess | WebFiddle | How.nz | Helix AI Assistant