{'fraction_non_alphanumeric': 0.014634146341463415, 'fraction_numerical': 0.003658536585365854, 'mean_word_length': 5.515873015873016, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Full Text Search of Web Archive Collections

The Internet Archive, in cooperation with the International Internet Preservation Consortium, is developing an open source full text search of Web archive collections. Web archive collection search presents the usual set of technical difficulties searching large collections of documents. It also introduces new challenges often at odds with typical search engine usage. This paper outlines the challenges and describes adaptation of an open source search engine, Nutch, to Web archive collection search. Statistics and observations indexing and searching small to medium-sized collections are presented. We close with a sketch of how we intend to tackle the main limitation, scaling archive collection search above the current ceiling of approximately 100 million documents.