title: Scraping a Website without Crawling the "Search Engine" section of the page
body: <p>Lately I've been trying to develop a Web Crawler using Scrapy to extract all the PDF documents from a specific domain and, although I can somehow achieve the goal, it still gets stuck in some kind of an "infinite loop" because of the "search engine" part of the said website. </p>
  
  <p>Is there any way that I can avoid this kind of behavior?</p>
  
  <p>Here's the code of the crawler that has been written so far (sorry if it's spaghetti, still trying to get better..):</p>
  
  <pre><code>class AirbusSpider(CrawlSpider):
      name = "test"
      tag = "aircraft"
      n_iterations = 3
      allowed_domains = ['airbus.com']
  
      start_urls = [
          'https://www.airbus.com'
      ]
  
  def start_requests(self):
      for url in self.start_urls:
          yield scrapy.Request(url=url, callback=self.parse)
  
  def parse(self, response):
      yield scrapy.Request(url=response.url+'/'+self.tag+'.html', callback=self.parse_item, meta={'iteration': self.n_iterations})
  
  
  def parse_item(self, response):
      current_iteration = response.meta['iteration']
      if(current_iteration&gt;0):
  
          temp_links = []
          valuable_links = []
          current_iteration-=1
  
          #all "somewhat-valuable" links to crawl into
          links = response.selector.xpath('//a[contains(@href,"aircraft")]/@href').extract() #all hrefs of current page
  
          for link in links:
              valuable_links.append(link)
  
          temp_links = set(valuable_links) 
          valuable_links = list(temp_links)
  
          for link in valuable_links:
              link = str(link.encode('utf-8'))
              if '.pdf' not in link:
                  if(link[0] == '/' or link[0] == '#'):
                      yield scrapy.Request(url=self.start_urls[0]+link, callback=self.parse_item, meta={'iteration': current_iteration})
                  else:
                      yield scrapy.Request(url=link, callback=self.parse_item, meta={'iteration': current_iteration})
              else:
                  if(link[0] == '/'  or link[0] == '#'):
                      #increase the priority of these requests
                      yield scrapy.Request(url=self.start_urls[0]+link, callback=self.download_file, priority = 1)
                  else:
                      yield scrapy.Request(url=link, callback=self.download_file, priority = 1)
  
  def download_file(self, response):
      item = PdfLink()
      item['link'] = str(response.url)
  
      return item
  </code></pre>
  
  <p>I tried to limit the recursion by adding a cap (n_iterations) of how deeply the crawler should dig into. It works but still, I'd like to filter the links it crawls into in a proper manner.</p>
  
  <p>Thank you so much for helping me out, have a good week!</p>