title: Getting "Too many requests" error when scraping a particular website using scrapy
body: <p>I have written a spider to fetch details from <a href="http://allevents.in" rel="nofollow noreferrer">http://allevents.in</a>.
  Every time I tried to scrap, I am getting a response body </p>
  
  <pre><code>Too many requests, please try after some time or report this problem at contact@allevents.in
  </code></pre>
  
  <p>I tried using the shell command also.</p>
  
  <pre><code> scrapy shell 'http://allevents.in/new%20delhi/all'
  </code></pre>
  
  <p>But still I am getting the same response for <code>response.body</code>.
  I have tried other websites like <a href="http://www.amazon.in" rel="nofollow noreferrer">amazon</a> which works fine.
  Also the above url can be fetched using <code>requests</code> as well as <code>urllib.urlopen()</code>.</p>
  
  <p>Here is my <code>settings.py</code> file</p>
  
  <pre><code># -*- coding: utf-8 -*-
  
  # Scrapy settings for tutorial project
  #
  # For simplicity, this file contains only settings considered important or
  # commonly used. You can find more settings consulting the documentation:
  #
  #     http://doc.scrapy.org/en/latest/topics/settings.html
  #     http://scrapy.readthedocs.org/en/latest/topics/downloader-middleware.html
  #     http://scrapy.readthedocs.org/en/latest/topics/spider-middleware.html
  
  BOT_NAME = 'tutorial'
  
  SPIDER_MODULES = ['tutorial.spiders']
  NEWSPIDER_MODULE = 'tutorial.spiders'
  
  
  # Crawl responsibly by identifying yourself (and your website) on the user-agent
  #USER_AGENT = 'tutorial (+http://www.yourdomain.com)'
  
  # Obey robots.txt rules
  ROBOTSTXT_OBEY = True
  
  # Configure maximum concurrent requests performed by Scrapy (default: 16)
  # CONCURRENT_REQUESTS = 1
  
  # Configure a delay for requests for the same website (default: 0)
  # See http://scrapy.readthedocs.org/en/latest/topics/settings.html#download-delay
  # See also autothrottle settings and docs
  DOWNLOAD_DELAY = 5
  # The download delay setting will honor only one of:
  CONCURRENT_REQUESTS_PER_DOMAIN = 1
  CONCURRENT_REQUESTS_PER_IP = 1
  
  # Disable cookies (enabled by default)
  COOKIES_ENABLED = False
  
  # Disable Telnet Console (enabled by default)
  # TELNETCONSOLE_ENABLED = False
  
  # Override the default request headers:
  # DEFAULT_REQUEST_HEADERS = {
  #   'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
  #   'Accept-Language': 'en',
  # }
  
  # Enable or disable spider middlewares
  # See http://scrapy.readthedocs.org/en/latest/topics/spider-middleware.html
  #SPIDER_MIDDLEWARES = {
  #    'tutorial.middlewares.TutorialSpiderMiddleware': 543,
  #}
  
  # Enable or disable downloader middlewares
  # See http://scrapy.readthedocs.org/en/latest/topics/downloader-middleware.html
  # DOWNLOADER_MIDDLEWARES = {
  # #    'tutorial.middlewares.MyCustomDownloaderMiddleware': 543,
  #      'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': None,
  #      # 'tutorial.middlewares.ProxyMiddleware': 100,
  # }
  
  # Enable or disable extensions
  # See http://scrapy.readthedocs.org/en/latest/topics/extensions.html
  #EXTENSIONS = {
  #    'scrapy.extensions.telnet.TelnetConsole': None,
  #}
  
  # Configure item pipelines
  # See http://scrapy.readthedocs.org/en/latest/topics/item-pipeline.html
  #ITEM_PIPELINES = {
  #    'tutorial.pipelines.TutorialPipeline': 300,
  #}
  
  # Enable and configure the AutoThrottle extension (disabled by default)
  # See http://doc.scrapy.org/en/latest/topics/autothrottle.html
  AUTOTHROTTLE_ENABLED = True
  # The initial download delay
  AUTOTHROTTLE_START_DELAY = 5
  # The maximum download delay to be set in case of high latencies
  AUTOTHROTTLE_MAX_DELAY = 60
  # The average number of requests Scrapy should be sending in parallel to
  # each remote server
  AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
  # Enable showing throttling stats for every response received:
  AUTOTHROTTLE_DEBUG = False
  
  # Enable and configure HTTP caching (disabled by default)
  # See http://scrapy.readthedocs.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings
  # HTTPCACHE_ENABLED = True
  # HTTPCACHE_EXPIRATION_SECS = 0
  # HTTPCACHE_DIR = 'httpcache'
  #HTTPCACHE_IGNORE_HTTP_CODES = []
  # HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage'
  </code></pre>
  
  <p>I am a beginner in scrapy. please help</p>