title: python logging with subprocesses: get ordered output as it appears on screen also on log file
body: <p>I have a python script (<em>script1.py</em>) which generates sub processes as follows:</p>
  
  <pre><code>print 'This is the main script'
  status=os.system(command)
  </code></pre>
  
  <p>Now I execute the script and redirect the stdoutput like this:</p>
  
  <pre><code>python script1.py &gt; log.txt
  </code></pre>
  
  <p>The file log.txt contains first the output of the subprocess command (called using os.system) and only <em>after</em> the string <em>'This is the main script'</em>. I would expect the <em>opposite order</em>!</p>
  
  <p>If I use subprocess instead of os.system the situation is still the same:</p>
  
  <pre><code>print 'This is the main script'
  p=subprocess.Popen(command,shell=True, stdout=None, stderr=subprocess.STDOUT)
  status=p.returncode
  </code></pre>
  
  <p>So how can I redirect the stdoutput to a file and make sure that all subprocesses write in that file in the proper order? Note that if <strong>I do not redirect</strong> the standard output, the order of the log messages is correct (as it appears on the screen)!</p>
  
  <p><strong>UPDATE</strong></p>
  
  <p>This fixes my problem (as described <a href="http://algorithmicallyrandom.blogspot.ch/2009/10/python-tips-and-tricks-flushing-stdout.html" rel="nofollow">here</a>):</p>
  
  <pre><code>sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0)
  </code></pre>
  
  <p>Alternatively one can also force the printed content to be flushed:</p>
  
  <pre><code>sys.stdout.flush()
  </code></pre>
  
  <p>however, I think this is not ideal as it should be done after each print command. So I used the first approach which has also the advantage that I can examine the logs in real time, without waiting that the computation ends before writing to file the buffered output.</p>