title: Encoding error in Python with Chinese characters
body: <p>I'm a beginner having trouble decoding several dozen CSV file with numbers + (Simplified) Chinese characters to UTF-8 in Python 2.7.</p>
  
  <p>I do not know the encoding of the input files so I have tried all the possible encodings I am aware of -- GB18030, UTF-7, UTF-8, UTF-16 &amp; UTF-32 (LE &amp; BE). Also, for good measure, GBK and GB3212, though these should be a subset of GB18030. The UTF ones all stop when they get to the first Chinese characters. The other encodings stop somewhere in the first line except GB18030. I thought this would be the solution because it read through the first few files and decoded them fine. Part of my code, reading line by line, is:</p>
  
  <pre><code>line = line.decode("GB18030")
  </code></pre>
  
  <p>The first 2 files I tried to decode worked fine. Midway through the third <strong>file</strong>, Python spits out</p>
  
  <pre><code>UnicodeDecodeError: 'gb18030' codec can't decode bytes in position 168-169: illegal multibyte sequence
  </code></pre>
  
  <p>In this file, there are about 5 such errors in about a million lines.</p>
  
  <p>I opened the input file in a text editor and checked which characters were giving the decoding errors, and the first few all had Euro signs in a particular column of the CSV files. I am fairly confident these are typos, so I would just like to delete the Euro characters. I would like to examine types of encoding errors one by one; I would like to get rid of all the Euro errors but do not want to just ignore others until I look at them first.</p>
  
  <p>Edit: I used <code>chardet</code> which gave GB2312 as the encoding with .99 confidence for all files. I tried using GB2312 to decode which gave: </p>
  
  <pre><code>UnicodeDecodeError: 'gb2312' codec can't decode bytes in position 108-109: illegal multibyte sequence
  </code></pre>