title: Python 3 Website detects scraper when using User-Agent spoofing
body: <p>I'm trying to scrape some information from Indeed.com using urllib.  Occasionally, the job link gets redirected to the hiring company's webpage.  When this happens, Indeed throws up some html about using an incompatible browser or device, rather than continuing to the redirected page.  After looking around, I found that in most cases spoofing urllib's user agent to look like a browser is enough to get around this, but this doesn't seem to be the case here.</p>
  
  <p>Any suggestions on where to go beyond spoofing the User-Agent?  Is it possible Indeed is able to realize the User-Agent is spoofed, and that there is no way around this?</p>
  
  <p>Here's an example of the code:</p>
  
  <pre><code>import urllib
  from fake_useragent import UserAgent
  from http.cookiejar import CookieJar
  
  ua = UserAgent()
  
  website = 'http://www.indeed.com/rc/clk?jk=0fd52fac51427150&amp;fccid=7f79c79993ec7e60'
  req = urllib.request.Request(website)
  cj = CookieJar()
  opener = urllib.request.build_opener(urllib.request.HTTPCookieProcessor(cj))
  opener.addheaders = [('User-Agent', ua.chrome)]
  response = opener.open(req)
  
  print(response.read().decode('utf-8'))
  </code></pre>
  
  <p>Thanks for the help!</p>