title: Processing a Django UploadedFile as UTF-8 with universal newlines
body: <p>In my django application, I provide a form that allows users to upload a file. The file can be in a variety of formats (Excel, CSV), come from a variety of platforms (Mac, Linux, Windows), and be encoded in a variety of encodings (ASCII, UTF-8).</p>
  
  <p>For the purpose of this question, let's assume that I have a view which is receiving <code>request.FILES['file']</code>, which is an instance of <code>InMemoryUploadedFile</code>, called <code>file</code>. My problem is that <code>InMemoryUploadedFile</code> objects (like <code>file</code>):</p>
  
  <ol>
  <li>Do not support UTF-8 encoding (I see a <code>\xef\xbb\xbf</code> at the beginning of the file, which as I understand is a flag meaning 'this file is UTF-8').</li>
  <li>Do not support universal newlines (which probably the majority of the files uploaded to this system will need).</li>
  </ol>
  
  <p>Complicating the issue is that I wish to pass the file in to the python <code>csv</code> module, which does not natively support Unicode. I will happily accept answers that avoid this issue - once I get django playing nice with UTF-8 I'm sure I can bludgeon <code>csv</code> into doing the same. (Similarly, please ignore the requirement to support Excel - I am waiting until CSV works before I tackle parsing Excel files.)</p>
  
  <p>I have tried using <code>StringIO</code>,<code>mmap</code>,<code>codec</code>, and any of a wide variety of ways of accessing the data in an <code>InMemoryUploadedFile</code> object. Each approach has yielded differing errors, none so far have been perfect. This shows some of the code that I feel came the closest:</p>
  
  <pre><code>import csv
  import codecs
  
  class CSVParser:
      def __init__(self,file):
          # 'file' is assumed to be an InMemoryUploadedFile object.
          dialect = csv.Sniffer().sniff(codecs.EncodedFile(file,"utf-8").read(1024))
          file.open() # seek to 0
          self.reader = csv.reader(codecs.EncodedFile(file,"utf-8"),
                                   dialect=dialect)
          try:
              self.field_names = self.reader.next()
          except StopIteration:
              # The file was empty - this is not allowed.
              raise ValueError('Unrecognized format (empty file)')
  
          if len(self.field_names) &lt;= 1:
              # This probably isn't a CSV file at all.
              # Note that the csv module will (incorrectly) parse ALL files, even
              # binary data. This will catch most such files.
              raise ValueError('Unrecognized format (too few columns)')
  
          # Additional methods snipped, unrelated to issue
  </code></pre>
  
  <p>Please note that I haven't spent too much time on the actual parsing algorithm so it may be wildly inefficient, right now I'm more concerned with getting encoding to work as expected.</p>
  
  <p>The problem is that the results are also not encoded, despite being wrapped in the Unicode <code>codecs.EncodedFile</code> file wrapper.</p>
  
  <p><strong>EDIT:</strong> It turns out, the above code does in fact work. <code>codecs.EncodedFile(file,"utf-8")</code> is the ticket. It turns out the reason I thought it didn't work was that the terminal I was using does not support UTF-8. Live and learn!</p>