subprocess – Spawn and communicate with additional processes.¶

Purpose:	Spawn and communicate with additional processes.
Python Version:	New in 2.4

The subprocess module provides a consistent interface to creating and working with additional processes. It offers a higher-level interface than some of the other available modules, and is intended to replace functions such as os.system, os.spawn*, os.popen*, popen2.* and commands.*. To make it easier to compare subprocess with those other modules, I will re-create the previous examples using the functions being replaced.

The subprocess module defines one class, Popen() and a few wrapper functions which use that class. Popen() takes several arguments to make it easier to set up the new process, and then communicate with it via pipes. I will concentrate on example code here; for a complete description of the arguments, refer to section 17.1.1 of the library documentation.

Note

The API is roughly the same, but the underlying implementation is slightly different between Unix and Windows. All of the examples shown here were tested on Mac OS X. Your mileage on a non-Unix OS will vary.

Running External Command¶

To run an external command without interacting with it, such as one would do with os.system(), Use the call() function.

import subprocess

# Simple command
subprocess.call(['ls', '-1'], shell=True)

$ python subprocess_os_system.py
__init__.py
__init__.pyc
index.rst
interaction.py
repeater.py
signal_child.py
signal_parent.py
subprocess_os_system.py
subprocess_pipes.py
subprocess_popen2.py
subprocess_popen3.py
subprocess_popen4.py
subprocess_popen_read.py
subprocess_popen_write.py
subprocess_shell_variables.py
subprocess_signal_parent_shell.py
subprocess_signal_setsid.py

And since we set shell=True, shell variables in the command string are expanded:

import subprocess

# Command with shell expansion
subprocess.call('ls -1 $HOME', shell=True)

$ python subprocess_shell_variables.py
%backup%~
Desktop
Devel
Documents
DownloadedApps
Downloads
Library
Logitech
Magazines
Movies
Music
Pictures
Public
Sites
bender-old
bin
browser - logitech
cfx
emacs
iPod
page-speed-images
page-speed-javascript
pip-log.txt
public_html
ssh_config.tar.gz
texlive
tmp
versioned_home_files

Working with Pipes¶

By passing different arguments for stdin, stdout, and stderr it is possible to mimic the variations of os.popen().

popen¶

Reading from the output of a pipe:

import subprocess

print '\nread:'
proc = subprocess.Popen(['echo', '"to stdout"'], 
                        shell=True, 
                        stdout=subprocess.PIPE,
                        )
stdout_value = proc.communicate()[0]
print '\tstdout:', repr(stdout_value)

$ python subprocess_popen_read.py

read:
        stdout: '\n'

Writing to the input of a pipe:

import subprocess

print '\nwrite:'
proc = subprocess.Popen(['cat', '-'],
                        shell=True,
                        stdin=subprocess.PIPE,
                        )
proc.communicate('\tstdin: to stdin\n')

$ python subprocess_popen_write.py
        stdin: to stdin

write:

popen2¶

Reading and writing, as with popen2:

import subprocess

print '\npopen2:'

proc = subprocess.Popen(['cat', '-'],
                        shell=True,
                        stdin=subprocess.PIPE,
                        stdout=subprocess.PIPE,
                        )
stdout_value = proc.communicate('through stdin to stdout')[0]
print '\tpass through:', repr(stdout_value)

$ python subprocess_popen2.py

popen2:
        pass through: 'through stdin to stdout'

popen3¶

Separate streams for stdout and stderr, as with popen3:

import subprocess

print '\npopen3:'
proc = subprocess.Popen('cat -; echo ";to stderr" 1>&2',
                        shell=True,
                        stdin=subprocess.PIPE,
                        stdout=subprocess.PIPE,
                        stderr=subprocess.PIPE,
                        )
stdout_value, stderr_value = proc.communicate('through stdin to stdout')
print '\tpass through:', repr(stdout_value)
print '\tstderr:', repr(stderr_value)

$ python subprocess_popen3.py

popen3:
        pass through: 'through stdin to stdout'
        stderr: ';to stderr\n'

popen4¶

Merged stdout and stderr, as with popen4:

import subprocess

print '\npopen4:'
proc = subprocess.Popen('cat -; echo ";to stderr" 1>&2',
                        shell=True,
                        stdin=subprocess.PIPE,
                        stdout=subprocess.PIPE,
                        stderr=subprocess.STDOUT,
                        )
stdout_value, stderr_value = proc.communicate('through stdin to stdout\n')
print '\tcombined output:', repr(stdout_value)

$ python subprocess_popen4.py

popen4:
        combined output: 'through stdin to stdout\n;to stderr\n'

Connecting Segments of a Pipe¶

By creating separate Popen instances and chaining their inputs and outputs together, you can create your own pipeline of commands just as with the Unix shell.

import subprocess

cat = subprocess.Popen(['cat', 'index.rst'], 
                        stdout=subprocess.PIPE,
                        )

grep = subprocess.Popen(['grep', '.. include::'],
                        stdin=cat.stdout,
                        stdout=subprocess.PIPE,
                        )

cut = subprocess.Popen(['cut', '-f', '3', '-d:'],
                        stdin=grep.stdout,
                        stdout=subprocess.PIPE,
                        )

end_of_pipe = cut.stdout

print 'Included files:'
for line in end_of_pipe:
    print '\t', line.strip()

$ python subprocess_pipes.py
Included files:
        subprocess_os_system.py
        subprocess_shell_variables.py
        subprocess_popen_read.py
        subprocess_popen_write.py
        subprocess_popen2.py
        subprocess_popen3.py
        subprocess_popen4.py
        subprocess_pipes.py
        repeater.py
        interaction.py
        signal_child.py
        signal_parent.py
        subprocess_signal_parent_shell.py
        subprocess_signal_setsid.py

Interacting with Another Command¶

All of the above examples assume a limited amount of interaction. The communicate() method reads all of the output and waits for child process to exit before returning. It is also possible to write to and read from the individual pipe handles used by the Popen instance. To illustrate this, I will use this simple echo program which reads its standard input and writes it back to standard output:

import sys

sys.stdout.write('repeater.py: starting\n')
sys.stdout.flush()

while True:
    next_line = sys.stdin.readline()
    if not next_line:
        break
    sys.stdout.write(next_line)
    sys.stdout.flush()

sys.stdout.write('repeater.py: exiting\n')
sys.stdout.flush()

Make note of the fact that repeater.py writes to stdout when it starts and stops. We can use that to show the lifetime of the subprocess in the next example. The following interaction example uses the stdin and stdout file handles owned by the Popen instance in different ways. In the first example, a sequence of 10 numbers are written to stdin of the process, and after each write the next line of output is read back. In the second example, the same 10 numbers are written but the output is read all at once using communicate().

import subprocess

print 'One line at a time:'
proc = subprocess.Popen('python repeater.py', 
                        shell=True,
                        stdin=subprocess.PIPE,
                        stdout=subprocess.PIPE,
                        )
for i in range(10):
    proc.stdin.write('%d\n' % i)
    output = proc.stdout.readline()
    print output.rstrip()
proc.communicate()

print
print 'All output at once:'
proc = subprocess.Popen('python repeater.py', 
                        shell=True,
                        stdin=subprocess.PIPE,
                        stdout=subprocess.PIPE,
                        )
for i in range(10):
    proc.stdin.write('%d\n' % i)

output = proc.communicate()[0]
print output

Notice where the “repeater.py: exiting” lines fall in the output for each loop:

$ python interaction.py
One line at a time:
repeater.py: starting
0
1
2
3
4
5
6
7
8

All output at once:
repeater.py: starting
0
1
2
3
4
5
6
7
8
9
repeater.py: exiting

Signaling Between Processes¶

In the article on os, I included an example of signaling between processes using os.fork() and os.kill(). Since each Popen instance provides a pid attribute with the process id of the child process, it is possible to do something similar with subprocess. For this example, I will again set up a separate script for the child process to be executed by the parent process.

import os
import signal
import time
import sys

pid = os.getpid()
received = False

def signal_usr1(signum, frame):
    "Callback invoked when a signal is received"
    global received
    received = True
    print 'CHILD %s: Received USR1' % pid
    sys.stdout.flush()

print 'CHILD %s: Setting up signal handler' % pid
sys.stdout.flush()
signal.signal(signal.SIGUSR1, signal_usr1)
print 'CHILD %s: Pausing to wait for signal' % pid
sys.stdout.flush()
time.sleep(3)

if not received:
    print 'CHILD %s: Never received signal' % pid

And now the parent process:

import os
import signal
import subprocess
import time
import sys

proc = subprocess.Popen(['python', 'signal_child.py'])
print 'PARENT: Pausing before sending signal...'
sys.stdout.flush()
time.sleep(1)
print 'PARENT: Signaling child'
sys.stdout.flush()
os.kill(proc.pid, signal.SIGUSR1)

And the output should look something like this:

$ python signal_parent.py
PARENT: Pausing before sending signal...
CHILD 14879: Setting up signal handler
CHILD 14879: Pausing to wait for signal
PARENT: Signaling child
CHILD 14879: Received USR1

Process Groups / Sessions¶

Because of the way the process tree works under Unix, if the process created by Popen spawns sub-processes, those children will not receive any signals sent to the parent. That means, for example, that it will be difficult to cause them to terminate by sending SIGINT or SIGTERM.

import os
import signal
import subprocess
import tempfile
import time
import sys

script = '''#!/bin/sh
echo "Shell script in process $$"
set -x
python signal_child.py
'''
script_file = tempfile.NamedTemporaryFile('wt')
script_file.write(script)
script_file.flush()

proc = subprocess.Popen(['sh %s' % script_file.name], shell=True, close_fds=True)
print 'PARENT: Pausing before sending signal to child %s...' % proc.pid
sys.stdout.flush()
time.sleep(1)
print 'PARENT: Signaling child %s' % proc.pid
sys.stdout.flush()
os.kill(proc.pid, signal.SIGUSR1)
time.sleep(3)

Notice that the pid used to send the signal is different from the pid of the child of the shell script waiting for the signal because in this example, there are three separate processes interacting:

subprocess_signal_parent_shell.py
a Unix shell process running the script created by the main python program
signal_child.py

$ python subprocess_signal_parent_shell.py
PARENT: Pausing before sending signal to child 14882...
Shell script in process 14882
+ python signal_child.py
CHILD 14883: Setting up signal handler
CHILD 14883: Pausing to wait for signal
PARENT: Signaling child 14882
CHILD 14883: Never received signal

The solution to this problem is to use a process group to associate the children so they can be signaled together. The process group is created with os.setsid(), setting the “session id” to the process id of the current process. All child processes inherit the session id, and since we only want it set in the shell created by Popen and its descendants we don’t call it in the parent process. Instead, we pass it to Popen as the preexec_fn argument so it is run inside the new process before it calls exec().

import os
import signal
import subprocess
import tempfile
import time
import sys

script = '''#!/bin/sh
echo "Shell script in process $$"
set -x
python signal_child.py
'''
script_file = tempfile.NamedTemporaryFile('wt')
script_file.write(script)
script_file.flush()

proc = subprocess.Popen(['sh %s' % script_file.name], 
                        shell=True, 
                        close_fds=True,
                        preexec_fn=os.setsid,
                        )
print 'PARENT: Pausing before sending signal to child %s...' % proc.pid
sys.stdout.flush()
time.sleep(1)
print 'PARENT: Signaling process group %s' % proc.pid
sys.stdout.flush()
os.killpg(proc.pid, signal.SIGUSR1)
time.sleep(3)

To signal the entire process group, we use os.killpg() with the pid value from our Popen instance.

$ python subprocess_signal_setsid.py
PARENT: Pausing before sending signal to child 14887...
Shell script in process 14887
+ python signal_child.py
CHILD 14888: Setting up signal handler
CHILD 14888: Pausing to wait for signal
PARENT: Signaling process group 14887
CHILD 14888: Received USR1

Conclusions¶

As you can see, subprocess can be much easier to work with than fork, exec, and pipes on their own. It provides all of the functionality of the other modules and functions it replaces, and more. The API is consistent for all uses and many of the extra steps of overhead needed (such as closing extra file descriptors, ensuring the pipes are closed, etc.) are “built in” instead of being handled by your code separately.

Doug Hellmann

Page Contents

Navigation

This Page