title: How do I decode unicode one line at a time in Python 2.7?
body: <p>The correct way to load unicode text from Python 2.7 is something like:</p>
  
  <pre><code>content = open('filename').read().decode('encoding'):
  for line in content.splitlines():
      process(line)
  </code></pre>
  
  <p>(<strong>Update</strong>: No it isn't. See the answers.)</p>
  
  <p>However, if the file is very large, I might want to read, decode and process it one line at a time, so that the whole file is never loaded into memory at once. Something like:</p>
  
  <pre><code>for line in open('filename'):
      process(line.decode('encoding'))        
  </code></pre>
  
  <p>The <code>for</code> loop's iteration over the open filehandle is a generator that reads one line at a time.</p>
  
  <p>This doesn't work though, because if the file is utf32 encoded, for example, then the bytes in the file (in hex) look something like:</p>
  
  <pre><code>hello\n = 68000000(h) 65000000(e) 6c000000(l) 6c000000(l) 6f000000(o) 0a000000(\n)
  </code></pre>
  
  <p>And the split into lines done by the <code>for</code> loop splits on the <code>0a</code> byte of the <code>\n</code> character, resulting in (in hex):</p>
  
  <pre><code>lines[0] = 0x 68000000 65000000 6c000000 6c000000 6f000000 0a
  lines[1] = 0x 000000
  </code></pre>
  
  <p>So part of the <code>\n</code> character is left at the end of line 1, and the remaining three bytes end up in line 2 (followed by whatever text is actually in line 2.) Calling <code>decode</code> on either of these lines understandably results in a <code>UnicodeDecodeError</code>.</p>
  
  <pre><code>UnicodeDecodeError: 'utf32' codec can't decode byte 0x0a in position 24: truncated data
  </code></pre>
  
  <p>So, obviously enough, splitting a unicode byte stream on <code>0a</code> bytes is not the correct way to split it into lines. Instead I should be splitting on occurrences of the full four-byte newline character (0x0a000000). However, I think the correct way to detect these characters is to decode the byte stream into a unicode string and look for <code>\n</code> characters - and this decoding of the full stream is exactly the operation I'm trying to avoid.</p>
  
  <p>This can't be an uncommon requirement. What's the correct way to handle it?</p>