title: Scrapy crawling speed is slow
body: <p>I had been crawling a ecommerce website using Scrapy and the crawling speed I am getting is 50~60 pages/minute.
  I am using 8 core linode server with 24 GB Ram, so I don't think network or server could be the reason.</p>
  
  <p>I am aware of the fact the some websites give download latency if we crawl them too often, but this is a large website and I am using <strong>proxymesh</strong> proxies too, so this also could not be possibly the reason.</p>
  
  <p>I think I am missing something in settings.py file -</p>
  
  <pre><code>BOT_NAME = 'scraper'
  
  SPIDER_MODULES = ['com.crawler.scraper.spiders']
  NEWSPIDER_MODULE = 'com.crawler.scraper.spiders'
  
  ITEM_PIPELINES = {
      'com.crawler.scraper.pipelines.generic_pipeline.Pipeline': 300
  }
  RANDOMIZE_DOWNLOAD_DELAY = False
  COOKIES_ENABLED = False
  CONCURRENT_ITEMS = 200
  CONCURRENT_REQUESTS=50
  DOWNLOAD_DELAY= 0
  CONCURRENT_REQUESTS_PER_DOMAIN=50
  LOG_LEVEL = 'ERROR'
  RETRY_TIMES = 2
  CONNECTION_TIMEOUT = 30
  USER_AGENT_LIST = 'user_agents.txt'
  
  DOWNLOADER_MIDDLEWARES = {
      'random_useragent.RandomUserAgentMiddleware': 400,
      'com.crawler.scraper.middlewares.proxy_middleware.ProxyMiddleware': 410,
      'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None,
  }
  </code></pre>
  
  <p>I started from <strong>DOWNLOAD_DELAY 5</strong> and now even I have kept it as 0 but still I couldn't see any significant change in the number of requests per minute.</p>
  
  <p>So, my questions are -></p>
  
  <p>1) What should I do to increase the speed to at least 200 requests/minute?
  2) What should be the ideal value for CONCURRENT_ITEMS?</p>