title: Scraping elements rendered using React JS with BeautifulSoup
body: <p>I want to scrape anchor links with class="_1UoZlX" from the search results from this particular page - <a href="https://www.flipkart.com/search?as=on&amp;as-pos=1_1_ic_sam&amp;as-show=on&amp;otracker=start&amp;page=6&amp;q=samsung+mobiles&amp;sid=tyy%2F4io" rel="noreferrer">https://www.flipkart.com/search?as=on&amp;as-pos=1_1_ic_sam&amp;as-show=on&amp;otracker=start&amp;page=6&amp;q=samsung+mobiles&amp;sid=tyy%2F4io</a></p>
  
  <p>When I created a soup from the page I realised that the search results are being rendered using React JS and hence I can't find them in the page source (or in the soup).</p>
  
  <p>Here's my code</p>
  
  <pre><code>import requests
  from bs4 import BeautifulSoup
  from selenium import webdriver
  from selenium.webdriver.common.by import By
  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.support import expected_conditions as EC
  
  
  listUrls = ['https://www.flipkart.com/search?as=on&amp;as-pos=1_1_ic_sam&amp;as-show=on&amp;otracker=start&amp;page=6&amp;q=samsung+mobiles&amp;sid=tyy%2F4iof']
  PHANTOMJS_PATH = './phantomjs'
  browser = webdriver.PhantomJS(PHANTOMJS_PATH)
  urls=[]
  
  for url in listUrls:
      browser.get(url)
      wait = WebDriverWait(browser, 20)
      wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, "_1UoZlX")))
      soup = BeautifulSoup(browser.page_source,"html.parser")
      results = soup.findAll('a',{'class':"_1UoZlX"})
      for result in results:
          link = result["href"]
          print link
          urls.append(link)
      print urls
  </code></pre>
  
  <p>This is the error I'm getting.</p>
  
  <pre><code>Traceback (most recent call last):
    File "fetch_urls.py", line 19, in &lt;module&gt;
      wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, "_1UoZlX")))
    File "/usr/local/lib/python2.7/site-packages/selenium/webdriver/support/wait.py", line 80, in until
      raise TimeoutException(message, screen, stacktrace)
  selenium.common.exceptions.TimeoutException: Message: 
  Screenshot: available via screen
  </code></pre>
  
  <p>Someone mentioned in <a href="https://stackoverflow.com/questions/34363121/cant-extract-the-text-and-find-all-by-beautifulsoup">this answer</a> that there is a way to use selenium to process the javascript on a page. Can someone elaborate on that? I did some googling but couldn't find an approach that works for this particular case.</p>