{'fraction_non_alphanumeric': 0.012875536480686695, 'fraction_numerical': 0.0, 'mean_word_length': 4.645161290322581, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
The Application of Sampling to the Design of Structural Analysis Web Crawlers

The growth of the World Wide Web (WWW) has seen it evolve into a rich information resource. It is constantly traversed with the aid of crawlers so as to harvest web content. When collecting data, crawlers have the potential of causing service denial to web servers. This paper proposes the application of sampling as a selection strategy in the design of structural analysis web crawlers. This has the benefit of alleviating the problems of bandwidth costs to web servers whilst retaining the quality of the data that is mined by crawlers. The initial results of this study are promising and are presented in this paper.