title: Scrapy Crawler Stops In The Middle Of Crawl
body: <p>I've run into an issue with a Scrapy spider. I scraping a list of urls from a txt file and, in the middle of the scrape, the scrapy spider keeps closing. 
  Here is my general scraper logic </p>
  
  <pre><code>class DomainArticlesSpider(BaseSpider):
      name = "domain_articles"
      allowed_domains = ["domain.com"]
      f = open("/domain_urls.txt")
      start_urls = f
      def parse:
      item = DomainItem()
      ...
      return item 
  </code></pre>
  
  <p>The spider will end with:</p>
  
  <pre><code>2014-10-08 21:18:31-0400 [domain_articles] ERROR: Error downloading &lt;GET url&gt;: TCP connection timed out: 110: Connection timed out.
  
  2014-10-08 21:18:31-0400 [domain_articles] INFO: Closing spider (finished)
  2014-10-08 21:18:31-0400 [domain_articles] INFO: Stored csv feed (6848 items) in: culture_metadata2.csv
  2014-10-08 21:18:31-0400 [domain_articles] INFO: Dumping Scrapy stats:
      {'downloader/exception_count': 36,
       'downloader/exception_type_count/twisted.internet.error.DNSLookupError': 30,
       'downloader/exception_type_count/twisted.internet.error.TCPTimedOutError': 6,
       'downloader/request_bytes': 3301700,
       'downloader/request_count': 7658,
       'downloader/request_method_count/GET': 7658,
       'downloader/response_bytes': 200927645,
       'downloader/response_count': 7622,
       'downloader/response_status_count/200': 6848,
       'downloader/response_status_count/301': 637,
       'downloader/response_status_count/302': 8,
       'downloader/response_status_count/404': 129,
       'finish_reason': 'finished',
       'finish_time': datetime.datetime(2014, 10, 9, 1, 18, 31, 575068),
       'item_scraped_count': 6848,
       'log_count/DEBUG': 14637,
       'log_count/ERROR': 13,
       'log_count/INFO': 42,
       'response_received_count': 6977,
       'scheduler/dequeued': 7658,
       'scheduler/dequeued/disk': 7658,
       'scheduler/enqueued': 7658,
       'scheduler/enqueued/disk': 7658,
       'start_time': datetime.datetime(2014, 10, 9, 0, 44, 5, 340444)}
  2014-10-08 21:18:31-0400 [domain_articles] INFO: Spider closed (finished)
  </code></pre>
  
  <p>Any idea what is causing the stop? I don't understand why it is ending. There are still urls in the urls.txt file. The spider starts working again when delete the urls up until the point the spider has reached and restart it. </p>