The Python Oracle

Is subprocess.Popen not thread safe?

Become part of the top 3% of the developers by applying to Toptal https://topt.al/25cXVn

--

Music by Eric Matyas
https://www.soundimage.org
Track title: Puzzle Game 2

--

Chapters
00:00 Question
02:12 Accepted answer (Score 15)
03:52 Thank you

--

Full question
https://stackoverflow.com/questions/2119...

Accepted answer links:
http://bugs.python.org/issue2320
http://bugs.python.org/issue1336
http://bugs.python.org/issue14548
[subprocess32]: http://pypi.python.org/pypi/subprocess32/

--

Content licensed under CC BY-SA
https://meta.stackexchange.com/help/lice...

--

Tags
#python #concurrency #threadsafety #subprocess #popen

#avk47



ACCEPTED ANSWER

Score 15


On Python 2.x, there are various race conditions affecting subprocess.Popen. (e.g. on 2.7 it disables & restores garbage collection to prevent various timing issues, but this is not thread-safe in itself). See e.g. http://bugs.python.org/issue2320, http://bugs.python.org/issue1336 and http://bugs.python.org/issue14548 for a few of the issues in this area.

A substantial revision to subprocess was made in Python 3.2 which addresses these (amongst other things, the fork & exec code is in a C module, rather than doing some reasonably involved Python code in the critical part between fork and exec), and is available backported to recent Python 2.x releases in the subprocess32 module. Note the following from the PyPI page: "On POSIX systems it is guaranteed to be reliable when used in threaded applications."

I can reproduce the occasional (about 25% for me) crashes of the code above, but after using import subprocess32 as subprocess I've not seen any failures in 100+ runs.

Note that subprocess32 (and Python 3.2+) default to close_fds=True, but with subprocess32 in place, I saw no failures even with close_fds=False (not that you should generally need that).