{'fraction_non_alphanumeric': 0.014840182648401826, 'fraction_numerical': 0.0091324200913242, 'mean_word_length': 5.355072463768116, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Using Specialized Knowledge in Automated Web Document Summarization

Automated text summarization is a natural language processing task to generate short, concise, and comprehensive descriptions of essential content of documents. This paper is going to describe some new features in an online automated web document summarization system used in Seven Tones Search Engine (Zheng and Erbach 2002, Zheng 2001), a search engine specialized in linguistics and languages. The main idea of this system is to use algorithms designed specifically for Web pages in a specific knowledge domain to improve the quality of summarization. In special, linguistics features should be very important to linguistics document. The documents are assumed either HTML or plain text. A good HTML parser will affect summarization quality very much although it is not a part of summarization algorithm.