title: Using subprocess.Popen for Process with Large Output
body: <p>I have some Python code that executes an external app which works fine when the app has a small amount of output, but hangs when there is a lot. My code looks like:</p>
  
  <pre><code>p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
  errcode = p.wait()
  retval = p.stdout.read()
  errmess = p.stderr.read()
  if errcode:
      log.error('cmd failed &lt;%s&gt;: %s' % (errcode,errmess))
  </code></pre>
  
  <p>There are comments in the docs that seem to indicate the potential issue. Under wait, there is:</p>
  
  <blockquote>
    <p>Warning: This will deadlock if the child process generates enough output to a <code>stdout</code> or <code>stderr</code> pipe such that it blocks waiting for the OS pipe buffer to accept more data. Use <code>communicate()</code> to avoid that.</p>
  </blockquote>
  
  <p>though under communicate, I see:</p>
  
  <p>Note The data read is buffered in memory, so do not use this method if the data size is large or unlimited.</p>
  
  <p>So it is unclear to me that I should use either of these if I have a large amount of data. They don't indicate what method I should use in that case.</p>
  
  <p>I do need the return value from the exec and do parse and use both the <code>stdout</code> and <code>stderr</code>.</p>
  
  <p>So what is an equivalent method in Python to exec an external app that is going to have large output?</p>