title: In Django, how to call a subprocess with a slow start-up time
body: <p>Suppose you're running Django on Linux, and you've got a view, and you want that view to return the data from a subprocess called <strong><em>cmd</em></strong> that operates on a file that the view creates, for example likeso:</p>
  
  <pre><code> def call_subprocess(request):
       response = HttpResponse()
  
       with tempfile.NamedTemporaryFile("W") as f:
           f.write(request.GET['data']) # i.e. some data
  
       # cmd operates on fname and returns output
       p = subprocess.Popen(["cmd", f.name], 
                     stdout=subprocess.PIPE, 
                     stderr=subprocess.PIPE)
  
       out, err = p.communicate()
  
       response.write(p.out) # would be text/plain...
       return response
  </code></pre>
  
  <p>Now, suppose <strong><em>cmd</em></strong> has a very slow start-up time, but a very fast operating time, and it does not natively have a daemon mode. I would like to improve the response-time of this view.</p>
  
  <p>I would like to make the whole system would run much faster by starting up a number of instances of <strong><em>cmd</em></strong> in a worker-pool, have them wait for input, and having <strong><em>call_process</em></strong> ask one of those worker pool processes handle the data.</p>
  
  <p>This is really 2 parts:</p>
  
  <p>Part 1. A function that calls <strong><em>cmd</em></strong> and <strong><em>cmd</em></strong> waits for input. This could be done with pipes, i.e.</p>
  
  <pre><code>def _run_subcmd():
      p = subprocess.Popen(["cmd", fname], 
          stdout=subprocess.PIPE, stderr=subprocess.PIPE)
  
      out, err = p.communicate()
      # write 'out' to a tmp file
      o = open("out.txt", "W")
      o.write(out)
      o.close()
      p.close()
      exit()
  
  def _run_cmd(data):
      f = tempfile.NamedTemporaryFile("W")
      pipe = os.mkfifo(f.name)
  
      if os.fork() == 0:
          _run_subcmd(fname)
      else:
          f.write(data)
  
      r = open("out.txt", "r")
      out = r.read()
      # read 'out' from a tmp file
      return out
  
  def call_process(request):
      response = HttpResponse()
  
      out = _run_cmd(request.GET['data'])
  
      response.write(out) # would be text/plain...
      return response
  </code></pre>
  
  <p>Part 2. A set of workers running in the background that are waiting on the data. i.e. We want to extend the above so that the subprocess is already running, e.g. when the Django instance initializes, or this <strong><em>call_process</em></strong> is first called, a set of these workers is created</p>
  
  <pre><code>WORKER_COUNT = 6
  WORKERS = []
  
  class Worker(object):
      def __init__(index):
          self.tmp_file = tempfile.NamedTemporaryFile("W") # get a tmp file name
          os.mkfifo(self.tmp_file.name)
          self.p = subprocess.Popen(["cmd", self.tmp_file], 
              stdout=subprocess.PIPE, stderr=subprocess.PIPE)
          self.index = index
  
      def run(out_filename, data):
          WORKERS[self.index] = Null # qua-mutex??
          self.tmp_file.write(data)
          if (os.fork() == 0): # does the child have access to self.p??
              out, err = self.p.communicate()
              o = open(out_filename, "w")
              o.write(out)
              exit()
  
          self.p.close()
          self.o.close()
          self.tmp_file.close()
          WORKERS[self.index] = Worker(index) # replace this one
          return out_file
  
      @classmethod
      def get_worker() # get the next worker
      # ... static, incrementing index 
  </code></pre>
  
  <p>There should be some initialization of workers somewhere, like this:</p>
  
  <pre><code>def init_workers(): # create WORKERS_COUNT workers
      for i in xrange(0, WORKERS_COUNT):
          tmp_file = tempfile.NamedTemporaryFile()
          WORKERS.push(Worker(i))
  </code></pre>
  
  <p>Now, what I have above becomes something likeso:</p>
  
  <pre><code>def _run_cmd(data):
       Worker.get_worker() # this needs to be atomic &amp; lock worker at Worker.index
  
       fifo = open(tempfile.NamedTemporaryFile("r")) # this stores output of cmd
  
       Worker.run(fifo.name, data)
       # please ignore the fact that everything will be
       # appended to out.txt ... these will be tmp files, too, but named elsewhere.
  
       out = fifo.read()
       # read 'out' from a tmp file
       return out
  
  
  def call_process(request):
       response = HttpResponse()
  
       out = _run_cmd(request.GET['data'])
  
       response.write(out) # would be text/plain...
       return response
  </code></pre>
  
  <p>Now, the questions:</p>
  
  <ol>
  <li><p>Will this work? (I've just typed this off the top of my head into StackOverflow, so I'm sure there are problems, but conceptually, will it work)</p></li>
  <li><p>What are the problems to look for?</p></li>
  <li><p>Are there better alternatives to this? e.g. Could threads work just as well (it's Debian Lenny Linux)? Are there any libraries that handle parallel process worker-pools like this?</p></li>
  <li><p>Are there interactions with Django that I ought to be conscious of?</p></li>
  </ol>
  
  <p>Thanks for reading! I hope you find this as interesting a problem as I do.</p>
  
  <p>Brian</p>