title: Getting all instances of a 404 error with scrapy
body: <p>I have Scrapy crawling my site, finding links with a 404 response and returning those to a JSON file. This works really well. </p>
  
  <p>However, I cannot figure out how to get all instances of that bad link since the duplication filter is catching these links and not retrying them. </p>
  
  <p>Since our site has thousands of pages, the sections are managed by multiple teams, I need to be able to create a report of bad links per section rather than finding one and doing a search a replace across the entire site. </p>
  
  <p>Any help would be greatly appreciated. </p>
  
  <p>My current crawler:</p>
  
  <pre><code>import scrapy
  from scrapy.spiders import CrawlSpider, Rule
  from scrapy.linkextractors.lxmlhtml import LxmlLinkExtractor
  from scrapy.item import Item, Field
  
  # Add Items for exporting to JSON
  class DevelopersLinkItem(Item):
      url = Field()
      referer = Field()
      link_text = Field()
      status = Field()
      time = Field()
  
  class DevelopersSpider(CrawlSpider):
      """Subclasses Crawlspider to crawl the given site and parses each link to JSON"""
  
      # Spider name to be used when calling from the terminal
      name = "developers_prod"
  
      # Allow only the given host name(s)
      allowed_domains = ["example.com"]
  
      # Start crawling from this URL
      start_urls = ["https://example.com"]
  
      # Which status should be reported
      handle_httpstatus_list = [404]
  
      # Rules on how to extract links from the DOM, which URLS to deny, and gives a callback if needed
      rules = (Rule(LxmlLinkExtractor(deny=([
          '/android/'])), callback='parse_item', follow=True),)
  
      # Called back to for each requested page and used for parsing the response
      def parse_item(self, response):
          if response.status == 404:
              item = DevelopersLinkItem()
              item['url'] = response.url
              item['referer'] = response.request.headers.get('Referer')
              item['link_text'] = response.meta.get('link_text')
              item['status'] = response.status
              item['time'] = self.now.strftime("%Y-%m-%d %H:%M")
  
              return item
  </code></pre>
  
  <p>I've tried a few custom dupe filters but ultimately none of them worked. </p>