The Python Oracle

Output garbled when launching multiple ssh-sessions with pseudo-tty (need remote process to exit when ssh disconnects/is killed)

--------------------------------------------------
Hire the world's top talent on demand or became one of them at Toptal: https://topt.al/25cXVn
and get $2,000 discount on your first invoice
--------------------------------------------------

Music by Eric Matyas
https://www.soundimage.org
Track title: Over Ancient Waters Looping

--

Chapters
00:00 Output Garbled When Launching Multiple Ssh-Sessions With Pseudo-Tty (Need Remote Process To Exit Whe
02:18 Accepted Answer Score 2
04:17 Thank you

--

Full question
https://stackoverflow.com/questions/5902...

--

Content licensed under CC BY-SA
https://meta.stackexchange.com/help/lice...

--

Tags
#python #ssh

#avk47



ACCEPTED ANSWER

Score 2


I reproduce systematically the issue with the following script:

import subprocess
import time

if __name__ == "__main__":
    for i in range(0, 10):
        proc = subprocess.Popen(
            "ssh -tt -q localhost 'echo 11; echo 22; echo 33; '",
            shell=True
        )
    time.sleep(4)

And I think the issue is not related to Python. These multiple ssh with pseudo-TTY seem to conflict with each other's. Eventually, the terminal used to run this script ends up broken as well (whereas it wasn't sourced):

>cat test2.py
import subprocess
                 import time
                            import atexit
... etc ...

I checked the documentation and this -t option seems to do much more than what you are actually trying to achieve. When I remove the second t and the -q options, I sometimes (not often), get a cryptic error message stating that something went wrong (but I no longer manage to reproduce it). I checked with google but without much success. Still, I'm convinced that this option is overkill and I would rather focus on the undying processes. This one issue is well known:

Starting a process over ssh using bash and then killing it on sigint

The second answer is your -tt option, but the best answer suits your example very well and is superior (with -tt you solve the ssh propagation of the termination but do not tackle the same issue between Python and its subprocess). For example:

import subprocess
import time

if __name__ == "__main__":
    for i in range(0, 10):
        proc = subprocess.Popen(
            "ssh localhost 'sleep 90 & read ; kill $!'",
            shell=True,
            stdin=subprocess.PIPE
        )
    time.sleep(40)

With this solution, stdin is shared by all actors (python, the python subprocess, the ssh process, the sleep process), and its closure at any point in the chain is detected by the final business process, trigering a graceful shutdown.

Edit with locust: I gave it a quick try and the issue was that a simple 'kill' is ignored by the slave (looks like an issue on lucust side). It seems to work with a 'kill -9':

import subprocess
import time

if __name__ == "__main__":
    for i in range(0, 2):
        proc = subprocess.Popen(
            "ssh localhost 'python -m locust --slave --no-web -f ~devsup/users/flemaitre/tmp/locust_config.py & read ; kill -9 $!'",
            shell=True,
            stdin=subprocess.PIPE
        )
    time.sleep(40)