title: Debug while webcraping with scrapy?
body: <p>I am trying to webscrape a website with scrapy (with a 10s download delay + AUTOTHROTTLE_ENABLED = True + ROBOTSTXT_OBEY = True). When I run the command:</p>
  
  <pre><code>scrape crawl myspider -o mydata.csv
  </code></pre>
  
  <p>I get this output for multiple "lines" until 405:</p>
  
  <pre><code>[protego] DEBUG: Rule at line 4 without any user agent to enforce it on.
  </code></pre>
  
  <p>and also:</p>
  
  <pre><code>[scrapy.spidermiddlewares.httperror] INFO: Ignoring response &lt;405 https://www.funda.nl/koop/utrecht/&gt;: HTTP status code is not handled or not allowed
  </code></pre>
  
  <p>Why can't scrapy scrape this website? </p>
  
  <p>and this is the dumping stats (if useful): </p>
  
  <pre><code>2019-11-03 00:33:55 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
  {'downloader/request_bytes': 417,
   'downloader/request_count': 2,
   'downloader/request_method_count/GET': 2,
   'downloader/response_bytes': 23402,
   'downloader/response_count': 2,
   'downloader/response_status_count/405': 2,
   'elapsed_time_seconds': 13.42723,
   'finish_reason': 'finished',
   'finish_time': datetime.datetime(2019, 11, 2, 23, 33, 55, 12777),
   'httperror/response_ignored_count': 1,
   'httperror/response_ignored_status_count/405': 1,
   'log_count/DEBUG': 61,
   'log_count/INFO': 11,
   'memusage/max': 51818496,
   'memusage/startup': 51818496,
   'response_received_count': 2,
   'robotstxt/request_count': 1,
   'robotstxt/response_count': 1,
   'robotstxt/response_status_count/405': 1,
   'scheduler/dequeued': 1,
   'scheduler/dequeued/memory': 1,
   'scheduler/enqueued': 1,
   'scheduler/enqueued/memory': 1,
   'start_time': datetime.datetime(2019, 11, 2, 23, 33, 41, 585547)}
  </code></pre>