title: Crawl and monitor +1000 websites
body: <p>I need help defining an architecture for a tool that will scrape more than 1000 big websites daily for new updates.</p>
  
  <p>I'm planning to use Scrapy for this project:</p>
  
  <ul>
  <li>Giving that Scrapy needs a project for each website, how can I handle scraping 1000+ websites and storing it's data with Scrapy in just one project? I tried adding a project generator, but is this a good idea?</li>
  <li>How can I tell if a website was updated with new content so I can scrape it again?</li>
  </ul>
  
  <p>Thanks!</p>