title: Python - parsing html response
body: <p>In my code, a user inputs a search term and the <code>get_all_links</code> parses the html response and extract the links that start with ‘<em>http</em>’. When <code>req</code> is replaced with a hard coded url such as:</p>
  
  <pre><code>content = urllib.request.urlopen("http://www.ox.ac.uk")
  </code></pre>
  
  <p>The program returns a list of properly formatted links correctly. However passing in <code>req</code>, no links are returned. I suspect this may be a formatting blip.</p>
  
  <p>Here is my code:</p>
  
  <pre><code>import urllib.request
  def get_all_links(s): # function to get all the links
      d=0
      links=[] # getting all links into a list
      while d!=-1: # untill d is -1. i.e no links in that page
          d=s.find('&lt;a href=',d) # if &lt;a href is found
          start=s.find('"',d) # stsrt will be the next character
          end=s.find('"',start+1) # end will be upto "
          if d!=-1: # d is not -1
              d+=1  
              if(s[start+1]=='h'): # add the link which starts with http only.
                  links.append(s[start+1:end]) # to link list
      return links # return list
  
  def main():
      term = input('Enter a search term: ')
      url = 'http://www.google.com/search'
      value = {'q' : term}
      user_agent = 'Mozilla/5.0'
      headers = {'User-Agent' : user_agent}
      data = urllib.parse.urlencode(value)
      print(data)
      url = url + '?' + data
      print(url)
      req = urllib.request.Request(url, None, headers)
  
      content = urllib.request.urlopen(req)
      s = content.read()
      print(s)
      links = get_all_links(s.decode('utf-8'))
  
      for i in links: # print the returned list.
          print(i) 
  
  main()
  </code></pre>