title: python performance dealing with binary files
body: <p>I'm trying to convert a <a href="http://en.wikipedia.org/wiki/YCbCr" rel="nofollow">YCbCr-file</a> 
  from 8 bpp to 10 bpp.</p>
  
  <p>My best approach so far is still order of magnitude slower
  than the most basic naive C implementation.</p>
  
  <p>Naive approach in C, runs in about 8s. Making the code to
  work on chunks instead, drops the time to well under 1s.</p>
  
  <p>I'm curious about what kind of performance it's possible to get
  from standard python dealing with binary files. Example file is
  in <a href="http://en.wikipedia.org/wiki/Common_Intermediate_Format" rel="nofollow">CIF-resolution</a> and is "small" in comparison to content in 1080p.
  Feel free to add numpy-suggestions as well although I'm mainly interested
  in standard python.</p>
  
  <p>The test-file can be downloaded from</p>
  
  <pre><code>http://trace.eas.asu.edu/yuv/foreman/foreman_cif.7z
  </code></pre>
  
  <p><code>sha1sum</code> for correct 10-bit output is</p>
  
  <pre><code>c511dabc793383f7fd0ed69b4bb9b9f89ef73b84
  </code></pre>
  
  <p>python:
  </p>
  
  <pre><code>#!/usr/bin/env python
  
  import array
  
  f_in = 'foreman_cif.yuv'
  f_out = 'py_10bpp.yuv'
  
  def bytesfromfile(f):
      while True:
          raw = array.array('B')
          raw.fromstring(f.read(8192))
          if not raw:
              break
          yield raw
  
  with open(f_in, 'rb') as fd_in, \
          open(f_out, 'wb') as fd_out:
  
      for byte in bytesfromfile(fd_in):
          data = []
          for i in byte:
              i &lt;&lt;= 2
              data.append(i &amp; 0xff)
              data.append((i &gt;&gt; 8) &amp; 0xff)
  
          fd_out.write(array.array('B', data).tostring())
  </code></pre>
  
  <p>Naive C-dito:
  </p>
  
  <pre><code>#include &lt;stdio.h&gt;
  #include &lt;stdlib.h&gt;
  
  int main(int argc, char** argv)
  {
      int c;
      int d[2];
  
      FILE* fd_in;
      FILE* fd_out;
  
      fd_in = fopen("foreman_cif.yuv", "rb");
      fd_out = fopen("c_10bpp.yuv", "wb");
  
      while((c = fgetc(fd_in)) != EOF) {
          c &lt;&lt;= 2;
          d[0] = c &amp; 0xff;
          d[1] = (c &gt;&gt; 8) &amp; 0xff;
  
          fwrite(&amp;d[0], 1, 1, fd_out);
          fwrite(&amp;d[1], 1, 1, fd_out);
      }
  
      fclose(fd_in);
      fclose(fd_out);
  
      return EXIT_SUCCESS;
  }
  </code></pre>