Is it bad to use subprocess.call() or os.system() when writing Python shell scripts?

Using os.system() or subprocess.call(..., shell=True) can't replace a shell, because they actually use the shell.

os.system("foo") actually invokes sh -c "foo" -- that is to say, it runs foo as a shell script. Using this, then, is in no respect replacing a shell. This is also true in the exact same way for subprocess.call("foo", shell=True).

Using subprocess.Popen and functions from that family can replace a shell, but this often results in verbose and unwieldy code.

Consider the following shell script:

#!/bin/sh
foo "$1" | bar "$2"

Now, let's look at what it takes to reproduce that in Python in a way that doesn't start any shell under-the-hood:

#!/usr/bin/env python
import subprocess, sys

p1 = subprocess.Popen(["foo", sys.argv[1]], stdout=subprocess.PIPE)
p2 = subprocess.Popen(["bar", sys.argv[2]], stdin=p1.stdout)
sys.exit(p2.wait())

We went from 19 characters (after the shebang) to 148 characters (after the shebang and imports) -- and this was for a completely trivial script, one not using fancier features such as process substitution, command substitution, or the like.

In general it is not a bad thing to create another process from your own process. People do this constantly on the bash.

However, one always should ask oneself what is the best environment to do the task you need to do. For instance I could easily call a python script to cut (the linux tool) a column from a file. However, the overhead to first open the python interpreter, then save the output from cut, and then save that again is possibly higher than checking how to use the bash-tool with man.

However, collecting output from another "serious" program to do further calculations on that output, yes, you can do that nicely with subprocesses (though I would opt for storing that output in a file and then just read in the file if I need to rerun my script).

And this is where launching a subprocess may get tricky: depending on how you open a new subprocess, you can not rely anymore on environment variables. Especially when dealing with large input data, the output from the subprocess does not get piped further and therefore is collected in memory until the program finished, which might lead into a memory problem.

To put it short: if using python solves your problem faster than combining bash-only tools, sure, do it. If that involves launching serious subprocesses, ok. However, if you want to replace bash with python, do not do that.