title: Domain-specific crawling with different settings for each domain (e.g. speed) using Storm crawler
body: <p>I have discovered Storm crawler only recently and from the past experience and studies and work with different crawlers I find this project based on Apache Storm pretty robust and suitable for many use cases and scenarios.</p>
  
  <p>I have read some tutorials and tested the storm crawler with some basic setup. I would like to use the crawler in my project but there are certain things I am not sure if the crawler is capable of doing or even if it is suitable for such use cases.</p>
  
  <p>I would like to do small and large recursive crawls on many web domains with specific speed settings and limit to the number of fetched urls. The crawls can be started separately at any time with different settings (different speed, ignoring robots.txt for that domain, ignoring external links).</p>
  
  <p>Questions:</p>
  
  <ul>
  <li>Is the storm crawler suitable for such scenario?</li>
  <li>Can I set the limit to the maximum number of pages fetched by the
  crawler?</li>
  <li>Can I set the limits to the number of fetched pages for different
  domains?</li>
  <li>Can I monitor the progress of the crawl for specific domains separately?</li>
  <li>Can I set the settings dynamically without the need of uploading modified topology to storm?</li>
  <li>Is it possible to pause or stop crawling (for specific domain)?</li>
  <li>Is usually storm crawler running as one deployed topology?</li>
  </ul>
  
  <p>I assume that for some of these questions the answer may be in customizing or writing my own bolts or spouts. But I would rather avoid modifying Fetcher Bolt or main logic of the crawler as that would mean I am developing another crawler.</p>
  
  <p>Thank you.</p>