python subprocess with shell=True: redirections and platform-independent subprocess killing

Last time I was in a similar situation, I found out the easiest (and practically the only) solution was to kick off a thread which takes care of your child process. You can take different routes with this method, be it to parse the piping of the shell-style command and perform those in python code (which you said wasn't an option due to the blocking), which would at the same time fix your killing problem. Basically, thread encapsulation seems like the way to go.

Sadly my experience with subprocess is all on the Windows platform, which has tons of its own little quirks. subprocess has a lot of flaws all-around it seems, although it must do an okay job given the existence of the popen, popen2 and so forth modules that it is supposed to replace.

I too had a similar situation recently, I created a python subprocess that uses shell=True, and needed to kill it later. However, because of the shell param, the 'real' process is a child of the shell, i.e. a grandchild of the main process. Killing the child shell does not automatically kill off the grandchild.

In my top-level python script:

childProcess = subprocess.Popen('python other.py', shell=True)

To kill the shell and the 'real work' grandchild, I did this:

subprocess.call("ps -ef | awk '$3 == \"" + str(childProcess.pid) + "\" {print $2}' | xargs kill -9", shell=True)
childProcess.kill()

The first line kills off all children of the child process (based on ps -ef parentage, the second line kills off the child.

Interestingly, this is necessary on Ubuntu, but not strictly necessary on Mac OSX, as on the Mac the grandchild seems to take over the original child shell process id.

Answering some of the issues:

• If you want to deal with standard output and input from the command that is run as part of subprocess, you have to use "shell = true"

• Specific solution may depend on what you are trying to do. How ever, for processing output from a command that will run for a long time, there are some useful solutions

  • you could use poll, look at SO discussion: Getting realtime output using subprocess
  • Look at the following for asynchronous interaction with subprocess: http://code.activestate.com/recipes/576957-asynchronous-subprocess-using-asyncore/
  • The following augmentation of subprocess can also allow for async operations : http://code.google.com/p/subprocdev/
  • http://pypi.python.org/pypi/drainers/0.0.3
  • Other useful discussion on SO : python subprocess with timeout and large output (>64K)
  • As suggested, you could combine with the threads

• Popen instance provides a pid attribute with the process id of the child process as shell=True. You should be able to collect the command pid and use it with os.kill()

-

Going through your problems one at a time:

I'm basically forced to use shell=True, to get redirections to work

You can't just use the stdout and stderr parameters?

out_log = open("stdout_log", "w")
err_log = open("stderr_log", "w")
subproc = subprocess.popen(..., stdout=out_log, stderr=err_log, ...)

Processing the child's stdout/stderr in the parent python process is not an option, since I couldn't find functionality for doing it in a non-waiting way (and the parent python process must do other things while the child is running)

This is because of Windows. On Unix-type OSes, you just use the select module. Windows can only select on sockets, not files.

If I use shell=True then subprocess.kill() will only terminate the shell but not the child process

Because when shell=True the shell is the child process, and the commands are its children.

I'd need a reliable child process termination method that works on any platform (but at least linux and windows)

Does a reliable child process termination method even exist for Windows? Last I heard, even Task Manager's End Task wasn't 100% reliable. And in Linux, you might not be able to kill a process that has a file open via a crashed driver.

Like Stigma said, for Windows support you will need to use threads as subprocess proxies. Also, you should try to run with shell=False, and if you can't, please elaborate as to why not.