title: R web-scraping - hidden text in HTML
body: <p>I want to scrape the urls from the following page:</p>
  
  <p><a href="http://www.europarl.europa.eu/meps/en/1186/seeall.html?type=CRE&amp;leg=5" rel="nofollow">http://www.europarl.europa.eu/meps/en/1186/seeall.html?type=CRE&amp;leg=5</a></p>
  
  <p>There are 180 urls to be collected from this page (each is a link to a speech given in Parliament), but I am running into problems whenever there are more than 100 urls to be scraped, as the additional speeches are only accessible by clicking on the "See More" box at the bottom of the page. I've tried to figure out how to reveal the additional links that I think are hidden by the "getMore" function, but with no luck! Apologies for naiveté here...</p>
  
  <p>My current code is as follows:</p>
  
  <h2>Read in the page</h2>
  
  <pre><code>mep.speech.list.url &lt;-"http://www.europarl.europa.eu/meps/en/1186/seeall.html?type=CRE&amp;leg=5"
  speech.list.data&lt;-try(readLines(mep.speech.list.url),silent=TRUE)
  </code></pre>
  
  <h2>Find urls</h2>
  
  <pre><code>mep.speech.list&lt;-speech.list.data
  mep.speech.lines&lt;-grep("href",mep.speech.list)
  mep.speech.list&lt;-mep.speech.list[mep.speech.lines]
  mep.speech.lines&lt;-grep("target",mep.speech.list)
  mep.speech.list&lt;-mep.speech.list[mep.speech.lines]
  mep.speech.list&lt;-mep.speech.list[-length(mep.speech.list)]    
  </code></pre>
  
  <h2>Clean URLs</h2>
  
  <pre><code>mep.speech.list.end&lt;-regexpr("target",mep.speech.list)
  mep.speech.list&lt;-substr(mep.speech.list,1, mep.speech.list.end)
  mep.speech.list&lt;-gsub("\t","",mep.speech.list)
  mep.speech.list&lt;-gsub('&lt;a href=\"',"",mep.speech.list)
  mep.speech.list&lt;-gsub('\" target',"",mep.speech.list)
  mep.speech.list&lt;-gsub('\" targe',"",mep.speech.list)    
  mep.speech.list&lt;-gsub('\" targ',"",mep.speech.list)
  mep.speech.list&lt;-gsub('\" tar',"",mep.speech.list)
  mep.speech.list&lt;-gsub('\" ta',"",mep.speech.list)
  mep.speech.list&lt;-gsub('\" t',"",mep.speech.list)    
  mep.speech.list&lt;-mep.speech.list[5:length(mep.speech.list)]
  print(mep.speech.list)
  </code></pre>