title: How to concurrently crawl paginated webpages with unknown end?
body: <p>I'm trying to write a web crawler in Rust using the tokio asynchronous runtime. I want to fetch/process multiple pages asynchronously but I also want the crawler to stop when it reaches the end (in other words if there is nothing left to crawl). So far I have used <a href="https://docs.rs/futures/0.3.17/futures/future/fn.try_join_all.html" rel="nofollow noreferrer">futures::future::try_join_all</a> for getting a collective result from the async functions that I have provide as <code>Future</code>s but this obviously requires the program to know the total pages to crawl beforehand. For example:</p>
  <pre class="lang-rs prettyprint-override"><code>async fn fetch(_url: String) -&gt; Result&lt;String, ()&gt; {
      tokio::time::sleep(std::time::Duration::from_millis(100)).await;
  
      Ok(String::from(&quot;foo&quot;))
  }
  
  #[tokio::main]
  async fn main() {
      let search_url = &quot;https://example.com/?page={page_num}&quot;;
  
      let futures = (1..=3)
          .map(|page_num| search_url.replace(&quot;{page_num}&quot;, &amp;page_num.to_string()))
          .map(|url| fetch(url));
  
      let _ = futures::future::try_join_all(futures).await.unwrap();
  }
  </code></pre>
  <p><a href="https://play.rust-lang.org/?version=stable&amp;mode=debug&amp;edition=2018&amp;gist=9bafaa40dd139960748ed91eb25921d3" rel="nofollow noreferrer">Rust Playground</a></p>
  <p>In this simple example I have to know the total pages to go through (<code>1..=3</code>) before actually fetching them. What I want is, not providing any range and have a condition to stop the whole process. (e.g. if the HTML result contains &quot;not found&quot;)</p>
  <p>I looked into <a href="https://docs.rs/futures/0.3.17/futures/executor/fn.block_on.html" rel="nofollow noreferrer">futures::executor::block_on</a> but I'm not sure if it is something that I can utilize for this task.</p>