title: Python webscraping Javascript with Await
body: <p>I have a problem concerning webscraping with Python. I'm trying to get the data from the first table from <a href="https://www.nyse.com/ipo-center/filings" rel="nofollow noreferrer">https://www.nyse.com/ipo-center/filings</a> by using <code>from requests_html import AsyncHTMLSession</code>.</p>
  <p>My code is here:</p>
  <pre><code>from bs4 import BeautifulSoup
  from requests_html import AsyncHTMLSession
  
  #first define the URL and start the session
  url = 'http://www.nyse.com/ipo-center/filings'
  session = AsyncHTMLSession()
  
  #then get the URL content, and load the html content after parsing through the javascript
  r = await session.get(url)
  await r.html.arender()
  
  #then we create a beautifulsoup object based on the rendered html
  soup = BeautifulSoup(r.html.html, &quot;lxml&quot;)
  
  #then we find the first datatable, which is the one that contains upcoming IPO data
  table1 = soup.find('table', class_='table table-data table-condensed spacer-lg')
  </code></pre>
  <p>Now I have 2 problems with that:</p>
  <ol>
  <li>Oftentimes the website doesn't return any valid information from the <code>table1</code>, so I don't get the underlying information that's inside the table. So far I'm circumventing that by simply waiting a couple of seconds, and then run the loop again, until the dataframe is loaded. Probably not the best option though.</li>
  <li>The code does work within Jupyter Notebook, but once I upload it in .py format on my Server, I get the error message that <code>SyntaxError: 'await' outside async function</code>.</li>
  </ol>
  <p>Does anybody have a solution to the 2 problems mentioned above?</p>