title: Python, windows console and encodings (cp 850 vs cp1252)
body: <p>I thought I knew everything about encodings and Python, but today I came across a weird problem: although the console is set to code page 850 - and Python reports it correctly - parameters I put on the command line seem to be encoded in code page 1252. If I try to decode them with sys.stdin.encoding, I get the wrong result. If I assume 'cp1252', ignoring what sys.stdout.encoding reports, it works.</p>
  
  <p>Am I missing something, or is this a bug in Python ? Windows ? Note: I am running Python 2.6.6 on Windows 7 EN, locale set to French (Switzerland).</p>
  
  <p>In the test program below, I check that literals are correctly interpreted and can be printed - this works. But all values I pass on the command line seem to be encoded wrongly:</p>
  
  <pre><code>#!/usr/bin/python
  # -*- encoding: utf-8 -*-
  import sys
  
  literal_mb = 'utf-8 literal:   üèéÃÂç€ÈÚ'
  literal_u = u'unicode literal: üèéÃÂç€ÈÚ'
  print "Testing literals"
  print literal_mb.decode('utf-8').encode(sys.stdout.encoding,'replace')
  print literal_u.encode(sys.stdout.encoding,'replace')
  
  print "Testing arguments ( stdin/out encodings:",sys.stdin.encoding,"/",sys.stdout.encoding,")"
  for i in range(1,len(sys.argv)):
      arg = sys.argv[i]
      print "arg",i,":",arg
      for ch in arg:
          print "  ",ch,"-&gt;",ord(ch),
          if ord(ch)&gt;=128 and sys.stdin.encoding == 'cp850':
              print "&lt;-",ch.decode('cp1252').encode(sys.stdout.encoding,'replace'),"[assuming input was actually cp1252 ]"
          else:
              print ""
  </code></pre>
  
  <p>In a newly created console, when running</p>
  
  <pre><code>C:\dev&gt;test-encoding.py abcé€
  </code></pre>
  
  <p>I get the following output</p>
  
  <pre><code>Testing literals
  utf-8 literal:   üèéÃÂç?ÈÚ
  unicode literal: üèéÃÂç?ÈÚ
  Testing arguments ( stdin/out encodings: cp850 / cp850 )
  arg 1 : abcÚÇ
     a -&gt; 97
     b -&gt; 98
     c -&gt; 99
     Ú -&gt; 233 &lt;- é [assuming input was actually cp1252 ]
     Ç -&gt; 128 &lt;- ? [assuming input was actually cp1252 ]
  </code></pre>
  
  <p>while I would expect the 4th character to have an ordinal value of <strong>130</strong> instead of 233 (see the code pages <a href="http://en.wikipedia.org/wiki/Code_page_850" rel="noreferrer">850</a> and <a href="http://en.wikipedia.org/wiki/Windows-1252" rel="noreferrer">1252</a>). </p>
  
  <p><em>Notes: the value of 128 for the euro symbol is a mystery - since cp850 does not have it. Otherwise, the '?' are expected - cp850 cannot print the characters and I have used 'replace' in the conversions.</em></p>
  
  <p>If I change the code page of the console to 1252 by issuing <code>chcp 1252</code> and run the same command, I (correctly) obtain</p>
  
  <pre><code>Testing literals
  utf-8 literal:   üèéÃÂç€ÈÚ
  unicode literal: üèéÃÂç€ÈÚ
  Testing arguments ( stdin/out encodings: cp1252 / cp1252 )
  arg 1 : abcé€
     a -&gt; 97
     b -&gt; 98
     c -&gt; 99
     é -&gt; 233
     € -&gt; 128
  </code></pre>
  
  <p>Any ideas what I'm missing ?</p>
  
  <p><strong>Edit 1:</strong> I've just tested by reading sys.stdin. This works as expected: in cp850, typing 'é' results in an ordinal value of 130. So the problem is really for the command line only. So, is the command line treated differently than the standard input ?</p>
  
  <p><strong>Edit 2:</strong> It seems I had the wrong keywords. I found another very close topic on SO: <a href="https://stackoverflow.com/questions/846850/read-unicode-characters-from-command-line-arguments-in-python-2-x-on-windows">Read Unicode characters from command-line arguments in Python 2.x on Windows</a>. Still, if the command line is not encoded like sys.stdin, and since sys.getdefaultencoding() reports 'ascii', it seems there is no way to know its actual encoding. I find the answer using win32 extensions pretty hacky.</p>