The Python Oracle

CompletedProcess from subprocess.run() doesn't return a string

This video explains
CompletedProcess from subprocess.run() doesn't return a string

--

Become part of the top 3% of the developers by applying to Toptal
https://topt.al/25cXVn

--

Music by Eric Matyas
https://www.soundimage.org
Track title: Secret Catacombs

--

Chapters
00:00 Question
01:04 Accepted answer (Score 20)
02:48 Answer 2 (Score 5)
03:18 Answer 3 (Score 1)
03:59 Thank you

--

Full question
https://stackoverflow.com/questions/3409...

Question links:
[Python 3.5 docs]: https://docs.python.org/3/library/subpro...

Accepted answer links:
[Why is reading lines from stdin much slower in C++ than Python?]: https://stackoverflow.com/q/9371238/4279

Answer 3 links:
[here]: https://pymotw.com/2/subprocess/index.ht...

--

Content licensed under CC BY-SA
https://meta.stackexchange.com/help/lice...

--

Tags
#python #subprocess #python35

#avk47



ACCEPTED ANSWER

Score 23


proc.stdout is already a string in your case, run print(type(proc.stdout)), to make sure. It contains all subprocess' output -- subprocess.run() does not return until the child process is dead.

for text_line in proc.stdout: is incorrect: for char in text_string enumerates characters (Unicode codepoints) in Python, not lines. To get lines, call:

lines = result.stdout.splitlines()

The result may be different from .split('\n') if there are Unicode newlines in the string.

If you want to read the output line by line (to avoid running out of memory for long-running processes):

from subprocess import Popen, PIPE

with Popen(command, stdout=PIPE, universal_newlines=True) as process:
    for line in process.stdout:
        do_something_with(line)

Note: process.stdout is a file-like object in this case. Popen() does not wait for the process to finish -- Popen() returns immidiately as soon as the child process is started. process is a subprocess.Popen instance, not CompletedProcess here.

If all you need is to count the number of lines (terminated by b'\n') in the output, like wc -l:

from functools import partial

with Popen(command, stdout=PIPE) as process:
    read_chunk = partial(process.stdout.read, 1 << 13)
    line_count = sum(chunk.count(b'\n') for chunk in iter(read_chunk, b''))

See Why is reading lines from stdin much slower in C++ than Python?




ANSWER 2

Score 7


if you need to have STDOUT lines in an array to better manipulate them you simply miss to split output by the "Universal newline" separators

nmap_out = subprocess.run(args = ['nmap', '-T4', '-A', '192.168.1.128'],
                              universal_newlines = True,
                              stdout = subprocess.PIPE)

nmap_lines = nmap_out.stdout.splitlines()
print(nmap_lines)

output is:

['Starting Nmap 7.01 ( https://nmap.org ) at 2016-02-28 12:24 CET', 'Note: Host seems down. If it is really up, but blocking our ping probes, try -Pn', 'Nmap done: 1 IP address (0 hosts up) scanned in 2.37 seconds']



ANSWER 3

Score 1


You are seeing a string, compare:

import subprocess
proc = subprocess.run(
    args = [ 'cat', 'input.txt' ],
    universal_newlines = False,
    stdout = subprocess.PIPE)

print (type(proc.stdout))

class 'bytes'

run calls popen.communicate

communicate() returns a tuple (stdout_data, stderr_data). The data will be bytes or, if universal_newlines was True, strings.

Have a look here for more explanation and other shell interactions.