The Python Oracle

Is subprocess.Popen not thread safe?

--------------------------------------------------
Hire the world's top talent on demand or became one of them at Toptal: https://topt.al/25cXVn
and get $2,000 discount on your first invoice
--------------------------------------------------

Music by Eric Matyas
https://www.soundimage.org
Track title: City Beneath the Waves Looping

--

Chapters
00:00 Is Subprocess.Popen Not Thread Safe?
01:16 Accepted Answer Score 15
02:36 Thank you

--

Full question
https://stackoverflow.com/questions/2119...

--

Content licensed under CC BY-SA
https://meta.stackexchange.com/help/lice...

--

Tags
#python #concurrency #threadsafety #subprocess #popen

#avk47



ACCEPTED ANSWER

Score 15


On Python 2.x, there are various race conditions affecting subprocess.Popen. (e.g. on 2.7 it disables & restores garbage collection to prevent various timing issues, but this is not thread-safe in itself). See e.g. http://bugs.python.org/issue2320, http://bugs.python.org/issue1336 and http://bugs.python.org/issue14548 for a few of the issues in this area.

A substantial revision to subprocess was made in Python 3.2 which addresses these (amongst other things, the fork & exec code is in a C module, rather than doing some reasonably involved Python code in the critical part between fork and exec), and is available backported to recent Python 2.x releases in the subprocess32 module. Note the following from the PyPI page: "On POSIX systems it is guaranteed to be reliable when used in threaded applications."

I can reproduce the occasional (about 25% for me) crashes of the code above, but after using import subprocess32 as subprocess I've not seen any failures in 100+ runs.

Note that subprocess32 (and Python 3.2+) default to close_fds=True, but with subprocess32 in place, I saw no failures even with close_fds=False (not that you should generally need that).