title: How to use CrawlSpider from scrapy to click a link with javascript onclick?
body: <p>I want scrapy to crawl pages where going on to the next link looks like this:</p>
  
  <pre><code>&lt;a href="#" onclick="return gotoPage('2');"&gt; Next &lt;/a&gt;
  </code></pre>
  
  <p>Will scrapy be able to interpret javascript code of that?</p>
  
  <p>With <em>livehttpheaders</em> extension I found out that clicking Next generates a POST with a really huge piece of "garbage" starting like this:</p>
  
  <pre><code>encoded_session_hidden_map=H4sIAAAAAAAAALWZXWwj1RXHJ9n
  </code></pre>
  
  <p>I am trying to build my spider on the <code>CrawlSpider</code> class, but I can't really figure out how to code it, with <code>BaseSpider</code> I used the <code>parse()</code> method to process the first URL, which happens to be a login form, where I did a POST with:</p>
  
  <pre><code>def logon(self, response):
      login_form_data={ 'email': 'user@example.com', 'password': 'mypass22', 'action': 'sign-in' }
      return [FormRequest.from_response(response, formnumber=0, formdata=login_form_data, callback=self.submit_next)]
  </code></pre>
  
  <p>And then I defined submit_next() to tell what to do next. I can't figure out how do I tell CrawlSpider which method to use on the first URL?</p>
  
  <p>All requests in my crawling, except the first one, are POST requests. They are alternating two types of requests: pasting some data, and clicking "Next" to go to the next page.</p>