title: Being a good citizen and web-scraping
body: <p>I have a two part question.</p>
  
  <p>First, I'm writing a web-scraper based on the <a href="http://readthedocs.org/docs/scrapy/en/latest/topics/spiders.html#topics-spiders" rel="noreferrer">CrawlSpider</a> spider in Scrapy. I'm aiming to scrape a website that has many thousands (possible into the hundreds of thousands) of records. These records are buried 2-3 layers down from the start page. So basically I have the spider start on a certain page, crawl until it finds a specific type of record, and then parse the html. What I'm wondering is what methods exist to prevent my spider from overloading the site? Is there possibly a way to do thing's incrementally or put a pause in between different requests?</p>
  
  <p>Second, and related, is there a method with Scrapy to test a crawler without placing undue stress on a site? I know you can kill the program while it runs, but is there a way to make the script stop after hitting something like the first page that has the information I want to scrape?</p>
  
  <p>Any advice or resources would be greatly appreciated.</p>