title: Output data from subprocess command line by line
body: <p>I am trying to read a large data file (= millions of rows, in a very specific format) using a pre-built (in C) routine. I want to then yeild the results of this, line by line, via a generator function. </p>
  
  <p>I can read the file OK, but where as just running: </p>
  
  <p><code>&lt;command&gt; &lt;filename&gt;</code></p>
  
  <p>directly in linux will print the results line by line as it finds them, I've had no luck trying to replicate this within my generator function. It seems to output the entire lot as a single string that I need to split on newline, and of course then everything needs reading before I can yield line 1. </p>
  
  <p>This code will read the file, no problem: </p>
  
  <pre><code>import subprocess
  import config
  
  file_cmd = '&lt;command&gt; &lt;filename&gt;'
  
  for rec in (subprocess.check_output([file_cmd], shell=True).decode(config.ENCODING).split('\n')):
      yield rec
  
  </code></pre>
  
  <p>(ENCODING is set in config.py to iso-8859-1 - it's a Swedish site) </p>
  
  <p>The code I have works, in that it gives me the data, but in doing so, it tries to hold the whole lot in memory. I have larger files than this to process which are likely to blow the available memory, so this isn't an option. </p>
  
  <p>I've played around with bufsize on Popen, but not had any success (and also, I can't decode or split after the Popen, though I guess the fact I need to split right now is actually my problem!).</p>