title: Handling Cookies while scraping with Python
body: <p>I'm trying to scrape the links from the careers page on a college website, and I am getting this error.</p>
  
  <blockquote>
    <p>urllib.error.HTTPError: HTTP Error 302: The HTTP server returned a redirect error that would lead to an infinite loop.
    The last 30x error message was:
    Moved Temporarily</p>
  </blockquote>
  
  <p>I think this is because the site has a session cookie.  After doing a bit of reading, there seems to be many ways to get around this (Requests, http.cookiejar, Selenium/PhantomJs), but I don't know how to incorporate these solutions into my scraping program.</p>
  
  <p>This is my scraping program.  It's written in Python 3.6 with BeautifulSoup4.</p>
  
  <pre><code>from bs4 import BeautifulSoup
  from urllib.request import urlopen
  
  html = urlopen("https://jobs.fanshawec.ca/applicants/jsp/shared/search/SearchResults_css.jsp")
  soup = BeautifulSoup(html, 'html.parser')
  data = soup.select(".ft0 a")
  ads = []
  
  for i in data:
      link = i.get('href')
      ads.append(link)
  
  for job in ads:
      print(job)
      print('')
  </code></pre>
  
  <p>When I clear the cookies in my browser and manually go to the page I'm trying to scrape (<a href="https://jobs.fanshawec.ca/applicants/jsp/shared/search/SearchResults_css.jsp" rel="nofollow noreferrer">https://jobs.fanshawec.ca/applicants/jsp/shared/search/SearchResults_css.jsp</a>), I'm taken to a different page.  Once I have the cookie though, I can go directly to the SearchResults page that I want to scrape.</p>
  
  <p>This is the cookie:</p>
  
  <p><a href="https://i.stack.imgur.com/jiOUR.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/jiOUR.png" alt="This is the cookie"></a></p>
  
  <p>Any thoughts on how I can deal with this cookie?</p>