subprocess – Spawn and communicate with additional processes.¶

Purpose:	Spawn and communicate with additional processes.
Python Version:	New in 2.4

The subprocess module provides a consistent interface to creating and working with additional processes. It offers a higher-level interface than some of the other available modules, and is intended to replace functions such as os.system(), os.spawn*(), os.popen*(), popen2.*() and commands.*(). To make it easier to compare subprocess with those other modules, the examples here re-create those used for os and popen.

The subprocess module defines one class, Popen and a few wrapper functions which use that class. The constructor for Popen takes several arguments to make it easier to set up the new process, and then communicate with it via pipes. I will concentrate on example code here; for a complete description of the arguments, refer to section 17.1.1 of the library documentation.

Note

The API is roughly the same, but the underlying implementation is slightly different between Unix and Windows. All of the examples shown here were tested on Mac OS X. Your mileage on a non-Unix OS will vary.

Running External Command¶

To run an external command without interacting with it, such as one would do with os.system(), Use the call() function.

import subprocess

# Simple command
subprocess.call(['ls', '-1'], shell=True)

$ python subprocess_os_system.py
__init__.py
index.rst
interaction.py
repeater.py
signal_child.py
signal_parent.py
subprocess_os_system.py
subprocess_pipes.py
subprocess_popen2.py
subprocess_popen3.py
subprocess_popen4.py
subprocess_popen_read.py
subprocess_popen_write.py
subprocess_shell_variables.py
subprocess_signal_parent_shell.py
subprocess_signal_setsid.py

When shell is set to True, shell variables in the command string are expanded:

import subprocess

# Command with shell expansion
subprocess.call('ls -1 $HOME', shell=True)

$ python subprocess_shell_variables.py
%backup%~
Desktop
Devel
Documents
DownloadedApps
Downloads
Dropbox
Envs
Library
Logitech
Magazines
Movies
Music
Pictures
Public
Sites
bender-old
bin
browser - logitech
cfx
emacs
gnupg-old.tar.gz
iPod
page-speed-images
page-speed-javascript
pip-log.txt
public_html
ssh_config.tar.gz
texlive
tmp
trace.txt
versioned_home_files

Working with Pipes¶

By passing different arguments for stdin, stdout, and stderr it is possible to mimic the variations of os.popen().

popen¶

Reading from the output of a pipe:

import subprocess

print '\nread:'
proc = subprocess.Popen(['echo', '"to stdout"'], 
                        shell=True, 
                        stdout=subprocess.PIPE,
                        )
stdout_value = proc.communicate()[0]
print '\tstdout:', repr(stdout_value)

$ python -u subprocess_popen_read.py

read:
        stdout: '\n'

Writing to the input of a pipe:

import subprocess

print '\nwrite:'
proc = subprocess.Popen(['cat', '-'],
                        shell=True,
                        stdin=subprocess.PIPE,
                        )
proc.communicate('\tstdin: to stdin\n')

$ python -u subprocess_popen_write.py

write:
        stdin: to stdin

popen2¶

Reading and writing, as with popen2:

import subprocess

print '\npopen2:'

proc = subprocess.Popen(['cat', '-'],
                        shell=True,
                        stdin=subprocess.PIPE,
                        stdout=subprocess.PIPE,
                        )
stdout_value = proc.communicate('through stdin to stdout')[0]
print '\tpass through:', repr(stdout_value)

$ python -u subprocess_popen2.py

popen2:
        pass through: 'through stdin to stdout'

popen3¶

Separate streams for stdout and stderr, as with popen3:

import subprocess

print '\npopen3:'
proc = subprocess.Popen('cat -; echo ";to stderr" 1>&2',
                        shell=True,
                        stdin=subprocess.PIPE,
                        stdout=subprocess.PIPE,
                        stderr=subprocess.PIPE,
                        )
stdout_value, stderr_value = proc.communicate('through stdin to stdout')
print '\tpass through:', repr(stdout_value)
print '\tstderr:', repr(stderr_value)

$ python -u subprocess_popen3.py

popen3:
        pass through: 'through stdin to stdout'
        stderr: ';to stderr\n'

popen4¶

Merged stdout and stderr, as with popen4:

import subprocess

print '\npopen4:'
proc = subprocess.Popen('cat -; echo ";to stderr" 1>&2',
                        shell=True,
                        stdin=subprocess.PIPE,
                        stdout=subprocess.PIPE,
                        stderr=subprocess.STDOUT,
                        )
stdout_value, stderr_value = proc.communicate('through stdin to stdout\n')
print '\tcombined output:', repr(stdout_value)

$ python -u subprocess_popen4.py

popen4:
        combined output: 'through stdin to stdout\n;to stderr\n'

Connecting Segments of a Pipe¶

By creating separate Popen instances and chaining their inputs and outputs together, you can create your own pipeline of commands just as with the Unix shell.

import subprocess

cat = subprocess.Popen(['cat', 'index.rst'], 
                        stdout=subprocess.PIPE,
                        )

grep = subprocess.Popen(['grep', '.. include::'],
                        stdin=cat.stdout,
                        stdout=subprocess.PIPE,
                        )

cut = subprocess.Popen(['cut', '-f', '3', '-d:'],
                        stdin=grep.stdout,
                        stdout=subprocess.PIPE,
                        )

end_of_pipe = cut.stdout

print 'Included files:'
for line in end_of_pipe:
    print '\t', line.strip()

$ python -u subprocess_pipes.py
Included files:
        subprocess_os_system.py
        subprocess_shell_variables.py
        subprocess_popen_read.py
        subprocess_popen_write.py
        subprocess_popen2.py
        subprocess_popen3.py
        subprocess_popen4.py
        subprocess_pipes.py
        repeater.py
        interaction.py
        signal_child.py
        signal_parent.py
        subprocess_signal_parent_shell.py
        subprocess_signal_setsid.py

Interacting with Another Command¶

All of the above examples assume a limited amount of interaction. The communicate() method reads all of the output and waits for child process to exit before returning. It is also possible to write to and read from the individual pipe handles used by the Popen instance. A simple echo program that reads from standard input and writes to standard output illustrates this:

import sys

sys.stderr.write('repeater.py: starting\n')
sys.stderr.flush()

while True:
    next_line = sys.stdin.readline()
    if not next_line:
        break
    sys.stdout.write(next_line)
    sys.stdout.flush()

sys.stderr.write('repeater.py: exiting\n')
sys.stderr.flush()

Make note of the fact that repeater.py writes to stderr when it starts and stops. That information can be used to show the lifetime of the child process.

The next interaction example uses the stdin and stdout file handles owned by the Popen instance in different ways. In the first example, a sequence of 10 numbers are written to stdin of the process, and after each write the next line of output is read back. In the second example, the same 10 numbers are written but the output is read all at once using communicate().

import subprocess

print 'One line at a time:'
proc = subprocess.Popen('python repeater.py', 
                        shell=True,
                        stdin=subprocess.PIPE,
                        stdout=subprocess.PIPE,
                        )
for i in range(10):
    proc.stdin.write('%d\n' % i)
    output = proc.stdout.readline()
    print output.rstrip()
remainder = proc.communicate()[0]
print remainder

print
print 'All output at once:'
proc = subprocess.Popen('python repeater.py', 
                        shell=True,
                        stdin=subprocess.PIPE,
                        stdout=subprocess.PIPE,
                        )
for i in range(10):
    proc.stdin.write('%d\n' % i)

output = proc.communicate()[0]
print output

Notice where the "repeater.py: exiting" lines fall in the output for each loop:

$ python -u interaction.py
One line at a time:
repeater.py: starting
0
1
2
3
4
5
6
7
8
9
repeater.py: exiting


All output at once:
repeater.py: starting
repeater.py: exiting
0
1
2
3
4
5
6
7
8
9

Signaling Between Processes¶

The os examples include a demonstration of signaling between processes using os.fork() and os.kill(). Since each Popen instance provides a pid attribute with the process id of the child process, it is possible to do something similar with subprocess. For example, using this script for the child process to be executed by the parent process:

import os
import signal
import time
import sys

pid = os.getpid()
received = False

def signal_usr1(signum, frame):
    "Callback invoked when a signal is received"
    global received
    received = True
    print 'CHILD %s: Received USR1' % pid
    sys.stdout.flush()

print 'CHILD %s: Setting up signal handler' % pid
sys.stdout.flush()
signal.signal(signal.SIGUSR1, signal_usr1)
print 'CHILD %s: Pausing to wait for signal' % pid
sys.stdout.flush()
time.sleep(3)

if not received:
    print 'CHILD %s: Never received signal' % pid

and this parent process:

import os
import signal
import subprocess
import time
import sys

proc = subprocess.Popen(['python', 'signal_child.py'])
print 'PARENT: Pausing before sending signal...'
sys.stdout.flush()
time.sleep(1)
print 'PARENT: Signaling child'
sys.stdout.flush()
os.kill(proc.pid, signal.SIGUSR1)

the output will look something like:

$ python signal_parent.py
PARENT: Pausing before sending signal...
CHILD 65645: Setting up signal handler
CHILD 65645: Pausing to wait for signal
PARENT: Signaling child
CHILD 65645: Received USR1

Process Groups / Sessions¶

Because of the way the process tree works under Unix, if the process created by Popen spawns sub-processes, those children will not receive any signals sent to the parent. That means, for example, it will be difficult to cause them to terminate by sending SIGINT or SIGTERM.

import os
import signal
import subprocess
import tempfile
import time
import sys

script = '''#!/bin/sh
echo "Shell script in process $$"
set -x
python signal_child.py
'''
script_file = tempfile.NamedTemporaryFile('wt')
script_file.write(script)
script_file.flush()

proc = subprocess.Popen(['sh %s' % script_file.name], shell=True, close_fds=True)
print 'PARENT: Pausing before sending signal to child %s...' % proc.pid
sys.stdout.flush()
time.sleep(1)
print 'PARENT: Signaling child %s' % proc.pid
sys.stdout.flush()
os.kill(proc.pid, signal.SIGUSR1)
time.sleep(3)

Notice that the pid used to send the signal is different from the pid of the child of the shell script waiting for the signal because in this example, there are three separate processes interacting:

subprocess_signal_parent_shell.py
a Unix shell process running the script created by the main python program
signal_child.py

$ python subprocess_signal_parent_shell.py
PARENT: Pausing before sending signal to child 65648...
Shell script in process 65648
+ python signal_child.py
CHILD 65649: Setting up signal handler
CHILD 65649: Pausing to wait for signal
PARENT: Signaling child 65648
CHILD 65649: Never received signal

The solution to this problem is to use a process group to associate the children so they can be signaled together. The process group is created with os.setsid(), setting the “session id” to the process id of the current process. All child processes inherit the session id, and since we only want it set in the shell created by Popen and its descendants we don’t call it in the parent process. Instead, we pass it to Popen as the preexec_fn argument so it is run after the fork() inside the new process, before it calls exec().

import os
import signal
import subprocess
import tempfile
import time
import sys

script = '''#!/bin/sh
echo "Shell script in process $$"
set -x
python signal_child.py
'''
script_file = tempfile.NamedTemporaryFile('wt')
script_file.write(script)
script_file.flush()

proc = subprocess.Popen(['sh %s' % script_file.name], 
                        shell=True, 
                        close_fds=True,
                        preexec_fn=os.setsid,
                        )
print 'PARENT: Pausing before sending signal to child %s...' % proc.pid
sys.stdout.flush()
time.sleep(1)
print 'PARENT: Signaling process group %s' % proc.pid
sys.stdout.flush()
os.killpg(proc.pid, signal.SIGUSR1)
time.sleep(3)

To signal the entire process group, we use os.killpg() with the pid value from our Popen instance.

$ python subprocess_signal_setsid.py
PARENT: Pausing before sending signal to child 65652...
Shell script in process 65652
+ python signal_child.py
CHILD 65653: Setting up signal handler
CHILD 65653: Pausing to wait for signal
PARENT: Signaling process group 65652
CHILD 65653: Received USR1

Conclusions¶

As you can see, subprocess can be much easier to work with than fork, exec, and pipes on their own. It provides all of the functionality of the other modules and functions it replaces, and more. The API is consistent for all uses and many of the extra steps of overhead needed (such as closing extra file descriptors, ensuring the pipes are closed, etc.) are “built in” instead of being handled by your code separately.

Doug Hellmann

Page Contents

Navigation

This Page