title: Scraping infinite scrolling website with Selenium in Python
body: <p>I would like to scrap the content of this website that has scrolling:
  <a href="http://stocktwits.com/symbol/AAPL?q=AAPL" rel="nofollow noreferrer">http://stocktwits.com/symbol/AAPL?q=AAPL</a></p>
  
  <p>I found this answer for a similar question in Stactoverflow:
  <a href="https://stackoverflow.com/questions/12519074/scrape-websites-with-infinite-scrolling">scrape websites with infinite scrolling</a></p>
  
  <p>and here is the code copied from there:</p>
  
  <pre><code>from selenium import webdriver
  from selenium.webdriver.common.by import By
  from selenium.webdriver.common.keys import Keys
  from selenium.webdriver.support.ui import Select
  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.common.exceptions import TimeoutException
  from selenium.webdriver.support import expected_conditions as EC
  from selenium.common.exceptions import NoSuchElementException
  from selenium.common.exceptions import NoAlertPresentException
  import sys
  
  import unittest, time, re
  
  class Sel(unittest.TestCase):
      def setUp(self):
          self.driver = webdriver.Firefox()
          self.driver.implicitly_wait(30)
          self.base_url = "https://twitter.com"
          self.verificationErrors = []
          self.accept_next_alert = True
      def test_sel(self):
          driver = self.driver
          delay = 3
          driver.get(self.base_url + "/search?q=stckoverflow&amp;src=typd")
          driver.find_element_by_link_text("All").click()
          for i in range(1,100):
              self.driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
              time.sleep(4)
          html_source = driver.page_source
          data = html_source.encode('utf-8')
  
  
  if __name__ == "__main__":
      unittest.main()
  </code></pre>
  
  <p>Now instead of twitter I want to scrap Stocktwits website (the link is above).</p>
  
  <p>I modified the above code to this:</p>
  
  <pre><code>from selenium import webdriver
  from selenium.webdriver.common.by import By
  from selenium.webdriver.common.keys import Keys
  from selenium.webdriver.support.ui import Select
  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.common.exceptions import TimeoutException
  from selenium.webdriver.support import expected_conditions as EC
  from selenium.common.exceptions import NoSuchElementException
  from selenium.common.exceptions import NoAlertPresentException
  import sys
  
  import unittest, time, re
  
  class Sel(unittest.TestCase):
      def setUp(self):
          self.driver = webdriver.Firefox()
          self.driver.implicitly_wait(30)
          self.base_url = "http://stocktwits.com/symbol/AAPL?q=AAPL"
          self.verificationErrors = []
          self.accept_next_alert = True
      def test_sel(self):
          driver = self.driver
          delay = 3
          driver.get(self.base_url)
          driver.find_element_by_link_text("All").click()
          for i in range(1,100):
              self.driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
              time.sleep(4)
          html_source = driver.page_source
          data = html_source.encode('utf-8')
  
  
  if __name__ == "__main__":
      unittest.main()
  </code></pre>
  
  <p>But when I run the code I get this error:</p>
  
  <pre><code>NoSuchElementException: Message: Unable to locate element: {"method":"link text","selector":"All"}
  </code></pre>
  
  <p>I appreciate any help to find out what is wrong.</p>