title: How to ensure webpage elements load correctly after selenium execute_script call before scraping
body: <p>I have a project where I am attempting to scrape data from a database-type webpage, but find its load times are very unpredictable. The button clicks call JavaScript functions that change the buttons in the frame. I have developed a script that navigates through the scriptural book, then chapters, then verses, until it gets to the &quot;bottom&quot; level of the tree and returns information about which talks used that particular verse as a citation. The goal is to collect all the data about each verse cited, by whom, when, etc.</p>
  <p>My current focus is to just get all the data downloaded, but that is proving to be tricky. I have tried to use <code>browser.implicitly_wait</code> to get the behaviour I am wanting, and have also tried <code>WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.ID, &quot;citationindex&quot;))</code> and that hasn't seemed to work either.</p>
  <p>The current setup is shown below. It works as follows:</p>
  <ol>
  <li>Selenium opens the base url in Chrome and calls the recursive function</li>
  <li><code>dig_into_citations</code> is called on the browser which recursively starts search through the links in the 'citationindex' or 'citationindex2' element.</li>
  <li>The <code>get_citationindex</code> function is called first to determine which one the data is in, as it appears to switch back and forth, depending on what level of the tree you are at. If anyone knows why this is happening, that would be hugely helpful). This 'waits' for the citationindex element to load, which is determines by checking when link elements appear inside one of the two possible elements. Hacky but this seems to work most of the time.</li>
  <li>Then the logic in <code>dig_into_citations</code> checks to see if we are at the base level yet, which is where the talks appear. If so, I'll collect the data somehow (not worried about the details now) but am currently just printing out for debugging purposes.</li>
  <li>If not yet at the base level, then we are looking at either books, chapters, or verses and want to dig into each of those. I do this by running the javascript function within the <code>onclick</code> attribute of each of the button links (which has the form <code>getFilter('1', '101')</code> where the number of arguments depends on the level). I am passing the script through as well to store as the id for each talk.</li>
  </ol>
  <p>The problem I am having is that call to <code>execute_script</code> basically just updates the current webpage with new buttons, and it's not clear how long that takes each time, so I'm putting a 1 second sleep in before each iteration. I find anything slower causes it to choke.</p>
  <p>Like I say, I have tried the &quot;wait-until&quot; functionality in Selenium but it hasn't seemed to function as expected. I suspect that this is again because running the script in each case causes the elements inside the 'citationindex' (or 'citationindex2') function to update but the element itself is already there and already &quot;loaded&quot; so maybe that's why the wait-until stuff isn't working.</p>
  <p>Any insights into how this process could be improved and made robust would be greatly appreciated. If I am going about this in the wrong way, it would also help to know if there is a simpler approach to get at all the data I'm looking for rather than this rickety recursion through button click hell. Thanks in advance.</p>
  <pre><code>from bs4 import BeautifulSoup
  from selenium import webdriver
  import time
  
  
  def get_citationindex(browser):
      &quot;&quot;&quot;
      Gets the name of the citationindex element that contains the links of interest.
      &quot;&quot;&quot;
      CITATION_ID_NAMES = ['citationindex', 'citationindex2']
  
      # Gets the HTML associated with both citation index elements and attempts to find links inside
      def get_soup_list(browser):
          citation_id_elems = [browser.find_element_by_id(x).get_attribute('innerHTML') for x in CITATION_ID_NAMES]
          citation_id_soups = [BeautifulSoup(x, features=&quot;lxml&quot;) for x in citation_id_elems]
          soup_list = [soup for soup in citation_id_soups if soup.find_all('a')]
          return soup_list
  
      # This runs until the page has actually loaded and the links are found inside the proper citationindex element
      soup_list = []
      while not soup_list:
          soup_list = get_soup_list(browser)   # ! this hack is not great
  
      # the BeautifulSoup object of the citation index element is returned (i.e. for 'citationindex' or 'citationindex2')
      citationindex_soup = soup_list[0]
      return citationindex_soup
  
  
  def dig_into_citations(browser, script=''):
      &quot;&quot;&quot;
      Recursive function that digs into the citations (through book, chapter, and verse) until it find the talk references.
      It operates by calling the javascript functions on the page.
      &quot;&quot;&quot;
      soup = get_citationindex(browser)
  
      # the desired links are the only ones with 'div' tags
      links = [x for x in soup.find_all('a') if x.find('div')]
      talks = [x for x in links if &quot;getTalk&quot; in x.get('onclick')]
      header = soup.find(class_='volumetitle')
  
      if talks:
          for k in talks:
              ref = k.find(class_='reference').text
              title = k.find(class_='talktitle').text
              print(script, header.text, ref, title)
      else:
          for k in links:
              script = k.get('onclick')
              browser.execute_script(script)
              time.sleep(1)  # ! this hack is also not great
              dig_into_citations(browser, script)
  
  
  # Run a test case for one particular book
  browser = webdriver.Chrome()
  browser.implicitly_wait(20)
  browser.get('https://scriptures.byu.edu/#::fNYNY7267e401074')
  dig_into_citations(browser)
  </code></pre>
  <blockquote>
  <p>Note I am running this with Python 3.8 with <code>selenium=3.141.0</code> and <code>beautifulsoup4=4.9.3</code> using Selenium driver for Chrome 87</p>
  </blockquote>