title: HTML encoding and lxml parsing
body: <p>I'm trying to finally solve some encoding issues that pop up from trying to scrape HTML with lxml. Here are three sample HTML documents that I've encountered:</p>
  
  <p>1.</p>
  
  <pre><code>&lt;!DOCTYPE html&gt;
  &lt;html lang='en'&gt;
  &lt;head&gt;
     &lt;title&gt;Unicode Chars: 은 —’&lt;/title&gt;
     &lt;meta charset='utf-8'&gt;
  &lt;/head&gt;
  &lt;body&gt;&lt;/body&gt;
  &lt;/html&gt;
  </code></pre>
  
  <p>2.</p>
  
  <pre><code>&lt;!DOCTYPE html&gt;
  &lt;html xmlns="http://www.w3.org/1999/xhtml" xml:lang="ko-KR" lang="ko-KR"&gt;
  &lt;head&gt;
      &lt;title&gt;Unicode Chars: 은 —’&lt;/title&gt;
      &lt;meta http-equiv="content-type" content="text/html; charset=utf-8" /&gt;
  &lt;/head&gt;
  &lt;body&gt;&lt;/body&gt;
  &lt;/html&gt;
  </code></pre>
  
  <p>3.</p>
  
  <pre><code>&lt;?xml version="1.0" encoding="utf-8"?&gt;
  &lt;!DOCTYPE html
  PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"&gt;
  &lt;html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"&gt;
  &lt;head&gt;
      &lt;meta http-equiv="Content-Type" content="text/html; charset=utf-8" /&gt;
      &lt;title&gt;Unicode Chars: 은 —’&lt;/title&gt;
  &lt;/head&gt;
  &lt;body&gt;&lt;/body&gt;
  &lt;/html&gt;
  </code></pre>
  
  <p>My basic script:</p>
  
  <pre><code>from lxml.html import fromstring
  ...
  
  doc = fromstring(raw_html)
  title = doc.xpath('//title/text()')[0]
  print title
  </code></pre>
  
  <p>The results are:</p>
  
  <pre><code>Unicode Chars: ì ââ
  Unicode Chars: 은 —’
  Unicode Chars: 은 —’
  </code></pre>
  
  <p>So, obviously an issue with sample 1 and the missing <code>&lt;meta http-equiv="Content-Type" content="text/html; charset=utf-8" /&gt;</code> tag. The solution from <a href="https://stackoverflow.com/questions/2686709/encoding-in-python-with-lxml-complex-solution">here</a> will correctly recognize sample 1 as utf-8 and so it is functionally equivalent to my original code.</p>
  
  <p>The lxml docs appear conflicted:</p>
  
  <p>From <a href="http://lxml.de/elementsoup.html#using-only-the-encoding-detection" rel="nofollow noreferrer">here</a> the example seems to suggest we should use UnicodeDammit to encode the markup as unicode.</p>
  
  <pre><code>from BeautifulSoup import UnicodeDammit
  
  def decode_html(html_string):
      converted = UnicodeDammit(html_string, isHTML=True)
      if not converted.unicode:
          raise UnicodeDecodeError(
              "Failed to detect encoding, tried [%s]",
              ', '.join(converted.triedEncodings))
      # print converted.originalEncoding
      return converted.unicode
  
  root = lxml.html.fromstring(decode_html(tag_soup))
  </code></pre>
  
  <p>However <a href="http://lxml.de/parsing.html#python-unicode-strings" rel="nofollow noreferrer">here</a> it says:</p>
  
  <blockquote>
    <p>[Y]ou will get errors when you try [to parse] HTML data in a unicode string that specifies a charset in a meta tag of the header. You should generally avoid converting XML/HTML data to unicode before passing it into the parsers. It is both slower and error prone.</p>
  </blockquote>
  
  <p>If I try to follow the the first suggestion in the lxml docs, my code is now:</p>
  
  <pre><code>from lxml.html import fromstring
  from bs4 import UnicodeDammit
  ...
  dammit = UnicodeDammit(raw_html)
  doc = fromstring(dammit.unicode_markup)
  title = doc.xpath('//title/text()')[0]
  print title
  </code></pre>
  
  <p>I now get the following results:</p>
  
  <pre><code>Unicode Chars: 은 —’
  Unicode Chars: 은 —’
  ValueError: Unicode strings with encoding declaration are not supported.
  </code></pre>
  
  <p>Sample 1 now works correctly but sample 3 results in an error due to the <code>&lt;?xml version="1.0" encoding="utf-8"?&gt;</code> tag.</p>
  
  <p>Is there a correct way to handle all of these cases? Is there a better solution than the following?</p>
  
  <pre><code>dammit = UnicodeDammit(raw_html)
  try:
      doc = fromstring(dammit.unicode_markup)
  except ValueError:
      doc = fromstring(raw_html)
  </code></pre>