title: Using scrapy LinkExtractor with Selenium
body: <p>Is there a way to somehow use selenium to process javascript along with LinkExtractor? I want to crawl websites and pickup pdf files on the pages. Many of the PDFs are only available after javascript has been executed.</p>
  
  <pre><code>from scrapy.linkextractors import LinkExtractor
  from scrapy.spiders import CrawlSpider, Rule
  from scrapy.crawler import CrawlerProcess
  from scrapy.selector import Selector
  from selenium import webdriver 
  from scrapy.http import Request
  
  class Fetcher(CrawlSpider):
      name = "Fetcher"
      allowed_domains = ["example.com"]
      start_urls = ["http://www.example.com/"]
  
      rules = [Rule(LinkExtractor(allow=()), follow=True, callback='parse_item')]
  
      def __init__(self, *a, **kw):
          super(Fetcher, self).__init__(*a, **kw)
          self.driver = webdriver.PhantomJS
          self.links = open("links.txt", "a")
          self.pdfs = open("pdfs.txt", "a")
  
      def parse_start_url(self, response):
          #Do stuff
  
      def parse_item(self, response):
          #Do stuff
  
  process = CrawlerProcess({
      'USER_AGENT': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)'
  })
  
  process.crawl(Fetcher)
  process.start() # the script will block here until the crawling is finished
  </code></pre>