{'fraction_non_alphanumeric': 0.015473887814313346, 'fraction_numerical': 0.0, 'mean_word_length': 4.954022988505747, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Practical Issues of Crawling Large Web Collections

During large crawls of the Web, we have observed several anomalies in the implementation of the basic protocols by some Web sites. These anomalies impose costs on the design of a Web crawler and reduce the findability of information on the Web. We document several issues related to networking, DNS, HTTP, HTML and application programming. Our aim is to help Web crawler designers and Web application developers to improve the interoperability of their Web systems.