{'fraction_non_alphanumeric': 0.024743512371756187, 'fraction_numerical': 0.00724200362100181, 'mean_word_length': 5.186567164179104, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Studies on improving retrieval accuracy in web information retrieval

With the rapid spread of the Internet, anyone can easily access a variety of information on the World Wide Web (WWW). Since information resources on the WWW continue to grow, it has become increasingly difficult for users to find relevant information on the WWW. In this situation, Web search engines are one of the most popular methods for finding valuable information effectively. However, users are not satisfied with the retrieval accuracy of search engines. We consider that the following two facts cause this problem: (i) accurate indexing of Web pages by considering their contents is not performed, (ii) most search engines cannot provide search results that satisfy each user’s information need. The aim of this thesis is to address the above problems in order to improve retrieval accuracy in Web information retrieval. In information retrieval systems based on the vector space model, the TF-IDF scheme is widely used to characterize documents. However, in the case of documents with hyperlink structures such as Web pages, it is necessary to develop a technique for representing the contents of Web pages more accurately by exploiting the contents of their hyperlinked neighboring pages. Therefore, in order to address the problem (i), we propose several approaches to refining the TF-IDF scheme for a target Web page by using the contents of its hyperlinked neighboring pages. Experimental results show Doctor’s Thesis, Department of Information Systems, Graduate School of Information Science, Nara Institute of Science and Technology, NAIST-IS-DT0161021, February 6, 2004.