title: Python Requests Cookie Issues (Web Scraping)
body: <p>I am currently writing a web scraper that provides the data for my Python Flask server. My goal is to send POST request to the server and retrieve the JSON. My Flask API is currently hosted on a Heroku dyno which runs Linux. A while back, the server added cookies verification, which meant I had to send my POST requests with cookies. However, the server also added some sort of IP match very recently, which means I could not use the same cookie string on my Windows machine for the Heroku server. Here is what I am running on both machines:</p>
  <pre><code>import requests
  
  url = &quot;https://coursebook.utdallas.edu/clips/clip-coursebook.zog&quot; # POST endpoint
  
  payload = &quot;action=search&amp;s%5B%5D=ecs+3390&amp;s%5B%5D=term_20f&quot; # &quot;ecs 3390&quot; is the query
  headers = {
    'Connection': 'keep-alive',
    'Accept': '*/*',
    'DNT': '1',
    'X-Requested-With': 'XMLHttpRequest',
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.102 Safari/537.36',
    'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8',
    'Origin': 'https://coursebook.utdallas.edu',
    'Sec-Fetch-Site': 'same-origin',
    'Sec-Fetch-Mode': 'cors',
    'Sec-Fetch-Dest': 'empty',
    'Referer': 'https://coursebook.utdallas.edu/search',
    'Accept-Language': 'en-US,en;q=0.9,zh-CN;q=0.8,zh;q=0.7,zh-TW;q=0.6',
    'Cookie': 'PTGSESSID=052a6577047da9c404825cc393c56445'
  }
  
  response = requests.request(&quot;POST&quot;, url, headers=headers, data = payload)
  
  print(response.text.encode('utf8'))
  
  </code></pre>
  <p>On my Windows machine, I receive the response fine; however, when I run it on MacOS with the same code, it returns a blank response as the cookies do not match (you can try it with the same exact code); if I then go on that website on the Mac and manually get a cookie string from sending the request the normal way, and substitute it in the Python code, it now works on the Mac.</p>
  <p>However, that requires me to physically open a browser, go to the url, and click a button to send the POST request, then get the cookie string from the developer console. That is obviously not possible on Heroku dynos, which means I have been running this server from my raspberry pi for a while now.</p>
  <p>I have tried to fetch the landing page (<a href="https://coursebook.utdallas.edu/search" rel="nofollow noreferrer">https://coursebook.utdallas.edu/search</a>) where I can send a GET request and get the SET-COOKIES from the headers, but that cookie string doesn't seem to work which is strange. I am pretty sure that this should be pretty simple, since sending the request on the website does not seem too difficult at all. Apologies for the long post, but let me know if you have any ideas! My raspberry pi would thank you all for finally being able to shut down.</p>