Selection in Scale-Free Small World
  In this paper we compare the performance characteristics of our selection
based learning algorithm for Web crawlers with the characteristics of the
reinforcement learning algorithm. The task of the crawlers is to find new
information on the Web. The selection algorithm, called weblog update, modifies
the starting URL lists of our crawlers based on the found URLs containing new
information. The reinforcement learning algorithm modifies the URL orderings of
the crawlers based on the received reinforcements for submitted documents. We
performed simulations based on data collected from the Web. The collected
portion of the Web is typical and exhibits scale-free small world (SFSW)
structure. We have found that on this SFSW, the weblog update algorithm
performs better than the reinforcement learning algorithm. It finds the new
information faster than the reinforcement learning algorithm and has better new
information/all submitted documents ratio. We believe that the advantages of
the selection algorithm over reinforcement learning algorithm is due to the
small world property of the Web.
