title: How is data scraping based on location in Amazon?
body: <p>Whenever I want to scraping on amazon.com, I fail. Because Product information changes according to location in amazon.com</p>
  <p>This changing information is as follows;</p>
  <ul>
  <li>1-Price</li>
  <li>2-Shipping fee</li>
  <li>3-Customs fee</li>
  <li>4-Shipping status</li>
  </ul>
  <p>Changing the location with selenium is simple, but the processing speed is very slow. So That's why I need to scraping with scrapy or requests.</p>
  <p>However, although I imitate cookies and headers as in the browser, amazon.com does not allow me to change the location.</p>
  <p>There are two big problems.</p>
  <ol>
  <li>There is a data called &quot;ubid-main&quot;, I cannot derive a copy of
  this data. This is amazon without data. It does not allow to change
  location.</li>
  <li>Although I do the same for the header data, there is a difference
  between the outgoing data. Example: I use the exact same header in
  the browser. but in the browser the Content-Type goes as json, but
  in the code I made, it is text / html; charset = UTF-8 going.</li>
  </ol>
  <p>And it is very interesting that there is no information on this subject. You cannot do location-oriented scraping to the world's number one shopping site.</p>
  <p>Please enlighten me who knows the answer to this topic.
  If there is a solution as scrapy or requests, it is sufficient.
  Seriously, I haven't solved this issue for 1 year.</p>
  <pre><code>import requests
  from lxml import etree
  from random import choice
  from urllib3.exceptions import InsecureRequestWarning
  import urllib.parse
  import urllib3.request
  urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
  
      
  
  def location():
      headersdelivery = {
              'user-agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.190 Safari/537.36',
              'content-type':'application/x-www-form-urlencoded',
              'accept':'text/html,*/*',
              'x-requested-with':'XMLHttpRequest',
              'contenttype':'application/x-www-form-urlencoded;charset=utf-8',
              'origin':'https://www.amazon.com',
              'sec-fetch-site':'same-origin',
              'sec-fetch-mode':'cors',
              'sec-fetch-dest':'empty',
              'referer':'https://www.amazon.com/',
              'accept-encoding':'gzip, deflate, br',
              'accept-language':'tr-TR,tr;q=0.9,en-US;q=0.8,en;q=0.7'
              }
  
      payload = {
      'locationType':'LOCATION_INPUT',
      'zipCode':'34249',
      'storeContext':'generic',
      'deviceType':'web',
      'pageType':'Gateway',
      'actionSource':'glow',
      'almBrandId':'undefined'}
  
  
      sessionid = requests.session()
      url = &quot;https://www.amazon.com/gp/delivery/ajax/address-change.html&quot;
      ulkesecmereq = sessionid.post(url, headers=headersdelivery, data=payload,verify=False)
  
      return sessionid
  
  
  def response(locationsession):
      headers = {
      'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.190 Safari/537.36',
      'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
      'Accept-Language': 'en-US,en;q=0.5',
      'Accept-Encoding': 'gzip, deflate, br',
      'Connection': 'keep-alive',
      'Upgrade-Insecure-Requests': '1',
      'TE': 'Trailers'}
  
      postdata = {
      'storeContext':'generic',
      'pageType':'Gateway'
      }
      req = locationsession.post(&quot;https://www.amazon.com/gp/glow/get-location-label.html&quot;,headers=headers, data=postdata, verify=False)
      print(req.content)
  
  
  locationsession = location()
  response(locationsession)
  </code></pre>