title: Scraping yahoo finance comments in python with Selenium
body: <p>Brand new here and relatively new to python. I'm working on code to scrape only the comments from the yahoo finance conversation pages. I'm starting with the APPL page (<a href="https://finance.yahoo.com/quote/AAPL/community/" rel="nofollow noreferrer">https://finance.yahoo.com/quote/AAPL/community/</a>). I am using the inspect element function in the Edge browser and have located the comments. Example comment:</p>
  
  <pre><code> &lt;div class="C($c-fuji-grey-l) Mb(2px) Fz(14px) Lh(20px)" data-reactid="44"&gt;$175 tomorrow vote up or not vote down ! I vote up&lt;/div&gt;
  </code></pre>
  
  <p>I've written the following basic code to get just the first comment (I don't need the responses). I figure if I can successfully get one comment, I can then get them all. I realize I'll have to deal with the loading of additional comments, but one step at a time.</p>
  
  <pre><code>import requests
  from bs4 import BeautifulSoup
  
  page = requests.get("https://finance.yahoo.com/quote/AAPL/community/")
  soup = BeautifulSoup(page.content, 'html.parser')
  comm = soup.find(class_='C($c-fuji-grey-l) Mb(2px) Fz(14px) Lh(20px)')
  print(comm)
  </code></pre>
  
  <p>I keep getting "None" as the output. I don't believe javascript is involved here (I see script tags at the bottom of the HTML but not for the comments). I believe what I'm doing is too simple but I don't know how to proceed. How can I successfully scrape these comments? Definitely open to different solutions and thanks for reading.</p>
  
  <p><B><I><em>EDIT</em> 11-7-17</I></B></p>
  
  <p>I redid the program using selenium and have gotten the comments that exist on the page to appear. I'm now working on the following steps:</p>
  
  <p>1) Having the program click the drop down option for Newest Reactions before scraping anything.</p>
  
  <p>2) Having the program click the "Show More" button on the bottom of the page to render more comments for scraping. I want to scrape the comments over the last year but I don't see any true time stamps in the html, only "1 day ago", "two days ago", etc. I'm thinking that I could try to have the program keep clicking the "Show More" button until it reaches the first comment with a  time stamp of "1 year ago".</p>
  
  <p>3) Scraping the commenter username and comment date. Ideally I would like to have the commenter username, comment date, and comment saved in a file (csv or JSON is fine).</p>
  
  <p>My code is:</p>
  
  <pre><code> from selenium import webdriver
   driver = webdriver.Chrome("C:/YOUR_PATH_HERE/chromedriver.exe")
   driver.get("https://finance.yahoo.com/quote/AAPL/community/")
   timeout=10
   comments = driver.find_elements_by_xpath("//div[@class='C($c-fuji-grey-l) Mb(2px) Fz(14px) Lh(20px)']")
   coments = [x.text for x in comments]
   print(len(coments))
   print(coments)
  </code></pre>
  
  <p>As always, any help is appreciated.</p>