title: Scraping Amazon using BeautifulSoup on AWS Lambda
body: <p>I'm working on a project that requires me to scrape product titles/names from Amazon using AWS Lambda. My code is as follows:</p>
  <pre><code>import json
  from bs4 import BeautifulSoup
  from googleapiclient.discovery import build
  import requests
  import base64
  
  
  def lambda_handler(event, context):
   
      headers = {&quot;User-Agent&quot;: &quot;Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:79.0) Gecko/20100101 Firefox/79.0&quot;}
      URL = &quot;https://www.amazon.in/Amazon-Brand-Solimo-Foliage-Bedsheet/dp/B076ZTSW47/ref=sr_1_6_mod_primary_lightning_deal?dchild=1&amp;pd_rd_r=53f449ad-419e-466d-bab8-2a09a026efc6&amp;pd_rd_w=5XoS9&amp;pd_rd_wg=VXhi2&amp;pf_rd_p=d848f904-cfb3-4815-b7b2-fe0f44e4153f&amp;pf_rd_r=B9HGM8VTCBCPSTBNW5Z6&amp;qid=1616001168&amp;refinements=p_n_format_browse-bin%3A19560802031&amp;s=kitchen&amp;sbo=Tc8eqSFhUl4VwMzbE4fw%2Fw%3D%3D&amp;smid=AT95IG9ONZD7S&amp;sr=1-6&quot;
  
      content = requests.get(URL, headers = headers)
      soup = BeautifulSoup(content.text, 'html.parser')
      title = soup.find(&quot;span&quot;, attrs={&quot;id&quot;:'productTitle'}).string.strip()
  
      return {
           'title' : title,
       }
  </code></pre>
  <p>For some reason, this is only working when my URL is from Amazon.in (India) but not amazon.com (US) * Note: I am not in India (unsure if that plays a role in anything)</p>
  <p>If I keep all the rest same and simply change the URL to something from amazon.com (US). When I inspected both the .in and US pages, both had a span element with the id 'productTitle'
  for example:</p>
  <pre><code>URL = https://www.amazon.com/Saucony-Mens-Kinvara-Running-Shoe/dp/B07Q8Y4GQL/?_encoding=UTF8&amp;pd_rd_w=c0VM8&amp;pf_rd_p=de0c3fe6-321f-473e-bef6-6a700af423d3&amp;pf_rd_r=S7482G8JKWVS6GB5ADG8&amp;pd_rd_r=faa67bb1-ca9b-4c1d-a730-21c49cfd9b35&amp;pd_rd_wg=KiSz4&amp;ref_=pd_gw_trq_rep_sims_gw
  </code></pre>
  <p>I am getting this error:</p>
  <pre><code>Response
  {
   &quot;errorMessage&quot;: &quot;'NoneType' object has no attribute 'string'&quot;,
   &quot;errorType&quot;: &quot;AttributeError&quot;,
   &quot;stackTrace&quot;: [
     [
       &quot;/var/task/lambda_function.py&quot;,
       14,
       &quot;lambda_handler&quot;,
       &quot;title = soup.find(\&quot;span\&quot;, attrs={\&quot;id\&quot;:'productTitle'}).string.strip()&quot;
     ]
   ]
  </code></pre>
  <p>}</p>
  <p>I am pretty sure this has something to do with the User-Agent and Amazon disallowing scraping. I am very new to web-scraping so please let me know if I am doing something wrong or if there are any other changes to make. The code I have is incredibly simple so I am pretty sure the error is with the headers/User Agent but once again, I am very new to this and need some direction on it</p>
  <p><strong>EDIT: @MendelG <a href="https://stackoverflow.com/a/66678913/12349734">suggested an answer</a> that actually worked on other IDEs such as Pycharm or Spyder, but it is still giving the same error on AWS Lambda, Does anybody know the reason lambda is executing it differently?</strong></p>