title: How to speed up parsing an 10MB HTML file with BeautifulSoup
body: <p>I'm parsing large HTMl files with Beautifulsoup that range between 3 and 10MB. Unfortunately, 99% of the data is content that I want to parse. The file practically contains a small header, a few js scripts and then between 1,000 and 10,000 items. Each items consists of the following table rows:</p>
  
  <pre><code>&lt;tr class="New" id="content_id"&gt;
      &lt;td class="item" align="center"&gt; 
      &lt;/td&gt;&lt;td align="center"&gt;
           &lt;a onclick="somecode"&gt;&lt;img src="/images/sample.gif" alt="alttext" class="image"&gt;&lt;/a&gt;
      &lt;/td&gt;&lt;td style="astyle"&gt;[content1]&lt;/td&gt;&lt;td&gt;[content2]&lt;/td&gt;&lt;td&gt;[content3]&lt;/td&gt;&lt;td&gt;[content4]&lt;/td&gt;&lt;td&gt;[content5]&lt;/td&gt;&lt;td style="bstyle"&gt;[content6]&lt;/td&gt;&lt;td&gt;[content7]&lt;/td&gt;&lt;td&gt;[content8]&lt;/td&gt;&lt;td&gt;[content9]&lt;/td&gt;&lt;td&gt;[content10]&lt;/td&gt;&lt;td&gt;[content11]&lt;/td&gt;&lt;td&gt;&lt;/td&gt;&lt;td&gt;[content12]&lt;/td&gt;&lt;td&gt;[content13]&lt;/td&gt;&lt;td&gt;
           &lt;a href="somejs"&gt;[content14]&lt;/a&gt;
      &lt;/td&gt;&lt;td&gt;
           &lt;a href="somejs"&gt;[content15]&lt;/a&gt;
      &lt;/td&gt;&lt;td&gt;[content16]&lt;/td&gt;&lt;td&gt;
           &lt;a title="" href="somejs"&gt;[content16]&lt;/a&gt;
      &lt;/td&gt;&lt;td&gt;
           &lt;a title="" href="somejs"&gt;[content17]&lt;/a&gt;
      &lt;/td&gt;
  &lt;/tr&gt;
  </code></pre>
  
  <p>Note every [content] placeholder is relevant data that I need to parse.</p>
  
  <p>I have tried a variety of common optimizations such as a) using different parsers, b) using SoupStrainer, c) define encoding</p>
  
  <p>b) and c) practically have no effect when I log the time it takes. The different parsers have a significant impact. When I run the script below on a 1.5k list of items (comparably small list), I'm getting the following parsing times (I am running the experiment on a 2012 Mac Book Air):</p>
  
  <pre><code>#1653 items parsed in 15.5 seconds with lxml
  #xml takes 27 sec
  #html5lib takes 69 sec
  #html.parser takes 24 sec
  
  current = datetime.datetime.utcnow()
  strainer = SoupStrainer('table', attrs={'id':'contenttable'})
  soup = BeautifulSoup(html,'lxml',parse_only=strainer,from_encoding="UTF-8")
  print datetime.datetime.utcnow() - current
  </code></pre>
  
  <p><strong>Question: Besides what I have used so far, are there any tweaks I can use to dramatically shorten the parsing time?</strong></p>
  
  <p>So far I can only think of increasing CPU power.</p>