Take the 2-minute tour ×

I am trying to employ a Subprocess in Python for keeping an external script open in a Server-like fashion. The external script first loads a model. Once this is done, it accepts requests via STDIN and returns processed strings to STDOUT.

So far, I've tried

tokenizer = subprocess.Popen([tokenizer_path, '-l', lang_prefix], stdin=subprocess.PIPE, stdout=subprocess.PIPE)

However, I cannot use

out = self._tokenizer.stdout.readline()

in order to repeatedly process input_strings by means of the subprocess – out will just be empty, no matter if I use stdout.read() or stdout.readline(). However, it works when I close the stdin with tokenizer.stdin.close() before reading STDOUT, but this closes the subprocess, which is not what I want as I would have to reload the whole external script again before sending another request.

Is there any way to use a subprocess in a server-like fashion in python without closing and re-opening it?

share|improve this question
Try flushing the stdin after doing a write, using tokenizer.stdin.flush() –  onon15 Nov 28 '12 at 10:57
I have tried that already – without success, unfortunately... –  sam Nov 28 '12 at 11:00
You might need an extra thread to get this to work. Check the implementation of subprocess.Popen.communicate to see how that's done. –  larsmans Nov 28 '12 at 11:25
I did something similar where I had a deamon communicating with my main program via named pipes. It works well. Would you consider using named pipes rather? I will be happy to share my implementation –  Sheena Nov 28 '12 at 11:27
@sam: Here ya go pastebin.com/Sm7A6AsD –  Sheena Nov 28 '12 at 11:47
show 2 more comments

2 Answers

up vote 0 down vote accepted

Thanks to this Answer, I found out that a slave handle must be used in order to properly communicate with the subprocess:

master, slave = pty.openpty()
tokenizer = subprocess.Popen(script, shell=True stdin=subprocess.PIPE, stdout=slave)
stdin_handle = process.stdin
stdout_handle = os.fdopen(master)

Now, I can communicate to the subprocess without closing it via

stdout_handle.readline() #gets the processed input
share|improve this answer
add comment

Your external script probably buffers its output, so you only can read it in the father when the buffer in the child is flushed (which the child must do itself). One way to make it flush its buffers is probably closing the input because then it terminates in a proper fashion and flushes its buffers in the process.

If you have control over the external program (i. e. if you can patch it), insert a flushing after the output is produced.

Otherwise programs sometimes can be made to not buffer their output by attaching them to a pseudo-TTY (many programs, including the stdlib, assume that when their output is going to a TTY, no buffering is wished). But this is a bit tricky.

share|improve this answer
I actually just found a solution using Python's Pseudo-terminal utilities, thanks to this answer. –  sam Nov 28 '12 at 13:04
I still would prefer an explicit flushing of the co-processes buffers. –  Alfe Nov 28 '12 at 13:21
add comment

Your Answer


