title: UnicodeDecodeError: 'utf8' codec can't decode byte 0xc3 in position 34: unexpected end of data
body: <p>I'm trying to write a <strong>scraper</strong> , but I'm having issues with encoding. When I tried to copy the string I was looking for into my text file, <code>python2.7</code> told me it didn't recognize the encoding, despite no special characters. Don't know if that's useful info.</p>
  <p>My code looks like this:</p>
  <pre><code>from urllib import FancyURLopener
  import os
  
  class MyOpener(FancyURLopener): #spoofs a real browser on Window
     version = 'Mozilla/5.0 (Windows; U; Windows NT 5.1; it; rv:1.8.1.11) Gecko/20071127 Firefox/2.0.0.11'
  
  print &quot;What is the webaddress?&quot;
  webaddress = raw_input(&quot;8::&gt;&quot;)
  
  print &quot;Folder Name?&quot;
  foldername = raw_input(&quot;8::&gt;&quot;)
  
  if not os.path.exists(foldername):
      os.makedirs(foldername)
  
  def urlpuller(start, page):
     while page[start]!= '&quot;':
        start += 1
     close = start
     while page[close]!='&quot;':
        close += 1
     return page[start:close]
  
  myopener = MyOpener()
  
  response = myopener.open(webaddress)
  site = response.read()
  
  nexturl = ''
  counter = 0
  
  while(nexturl!=webaddress):
     counter += 1
     start = 0
     
     for i in range(len(site)-35):
         if site[i:i+35].decode('utf-8') == u'&lt;img id=&quot;imgSized&quot; class=&quot;slideImg&quot;':
           start = i + 40
           break
     else:
        print &quot;Something's broken, chief. Error = 1&quot;
     
     next = 0
     
     for i in range(start, 8, -1):
        if site[i:i+8] == u'&lt;a href=':
           next = i
           break
     else:
        print &quot;Something's broken, chief. Error = 2&quot;
     
     nexturl = urlpuller(next, site)
     
     myopener.retrieve(urlpuller(start,site),foldername+'/'+foldername+str(counter)+'.jpg')
  
  print(&quot;Retrieval of &quot;+foldername+&quot; completed.&quot;)
  </code></pre>
  <p>When I try to run it using the site I'm using, it returns the error:</p>
  <pre><code>Traceback (most recent call last):
    File &quot;yada/yadayada/Python/scraper.py&quot;, line 37, in &lt;module&gt;
      if site[i:i+35].decode('utf-8') == u'&lt;img id=&quot;imgSized&quot; class=&quot;slideImg&quot;':
    File &quot;/usr/lib/python2.7/encodings/utf_8.py&quot;, line 16, in decode
      return codecs.utf_8_decode(input, errors, True)
  UnicodeDecodeError: 'utf8' codec can't decode byte 0xc3 in position 34: unexpected end of data
  </code></pre>
  <p>When pointed at <a href="http://google.com" rel="nofollow noreferrer">http://google.com</a>, it worked just fine.</p>
  <pre><code>&lt;meta http-equiv=&quot;Content-Type&quot; content=&quot;text/html; charset=utf-8&quot;&gt;
  </code></pre>
  <p>but when I try to decode using utf-8, as you can see, it does not work.</p>
  <p>Any suggestions?</p>