Change Rate Estimation and Optimal Freshness in Web Page Crawling
  For providing quick and accurate results, a search engine maintains a local
snapshot of the entire web. And, to keep this local cache fresh, it employs a
crawler for tracking changes across various web pages. However, finite
bandwidth availability and server restrictions impose some constraints on the
crawling frequency. Consequently, the ideal crawling rates are the ones that
maximise the freshness of the local cache and also respect the above
constraints. Azar et al. 2018 recently proposed a tractable algorithm to solve
this optimisation problem. However, they assume the knowledge of the exact page
change rates, which is unrealistic in practice. We address this issue here.
Specifically, we provide two novel schemes for online estimation of page change
rates. Both schemes only need partial information about the page change
process, i.e., they only need to know if the page has changed or not since the
last crawled instance. For both these schemes, we prove convergence and, also,
derive their convergence rates. Finally, we provide some numerical experiments
to compare the performance of our proposed estimators with the existing ones
(e.g., MLE).
