{'fraction_non_alphanumeric': 0.02127659574468085, 'fraction_numerical': 0.014627659574468085, 'mean_word_length': 5.193415637860082, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Integrating external information sources to guide worldwide web information retrieval

Information retrieval in the worldwide web environment poses unique challenges. The most common approaches involve indexing, but indexes introduce centralization and can never be up-to-date. This paper advocates using external databases and information sources as guides for locating worldwide web information. This approach has been implemented in WEBFIND, a tool which discovers scientific papers made available on the web by their authors. The external information sources used by WEBFIND are MELVYL, the online University of California library catalog, and NETFIND, a service for finding email addresses. WEBFIND combines the information available from these services in order to find good starting points for searching for the papers that a user wants. At several stages in its operation, WEBFIND must solve instances of what we call the field matching problem. This problem is to determine whether or not two syntactic values are alternative representations of the same semantic entity. For example the addresses Dept. of Comput. Sci. and Eng., University of California, San Diego, 9500 Gilman Dr. Dept. 0114, La Jolla, CA 92093 and UCSD, Computer Science and Engineering Department, CA 92093-0114 do designate the same department. We propose a systematic algorithm for this problem that takes into account the many complexities such as abbreviations and missing subfields that are evident in the example above.