{'fraction_non_alphanumeric': 0.0297029702970297, 'fraction_numerical': 0.0029702970297029703, 'mean_word_length': 5.017857142857143, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Filtering for personal web information agents

Text filtering algorithms are a main component of a personalized Web information agent. Web information agents, such as WebMate [2] and Syskill & Webert [5], essentially combine text filtering and algorithms that generate the incoming stream of Web documents. However, these algorithms are subject to additional restrictions: negative feedback is considered an inconvenience to the user, the number of training samples is limited, and space is a consideration (thus, storing full text of all examples is discouraged). We examine two types of adaptive filtering that appear suited to our requirements. The first one is based on TFIDF and the second, which was used previously for a Web information agent [5], on a naive Bayes classifier (NBC). We present a new variant on NBC that does not require explicit negative feedback. In a study with TREC data, we find that, while all methods are capable of achieving good performance, they each incur a different penalty.