title: ruby 1.9: invalid byte sequence in UTF-8
body: <p>I'm writing a crawler in Ruby (1.9) that consumes lots of HTML from a lot of random sites.<br>
  When trying to extract links, I decided to just use <code>.scan(/href="(.*?)"/i)</code> instead of nokogiri/hpricot (major speedup). The problem is that I now receive a lot of "<code>invalid byte sequence in UTF-8</code>" errors.<br>
  From what I understood, the <code>net/http</code> library doesn't have any encoding specific options and the stuff that comes in is basically not properly tagged.<br>
  What would be the best way to actually work with that incoming data? I tried <code>.encode</code> with the replace and invalid options set, but no success so far...</p>