{'fraction_non_alphanumeric': 0.020491803278688523, 'fraction_numerical': 0.0, 'mean_word_length': 5.400749063670412, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Enabling Federated Search with Heterogeneous Search Engines Combining FAST Data Search and Lucene

This report analyses Federated Search in the VASCODA context, specifically focusing on the existing TIB Hannover and UB Bielefeld search infrastructures. We first describe general requirements for a seamless integration of the two full-text search systems FAST (Bielefeld) and Lucene (Hannover), and evaluate possible scenarios, types of queries, and different ranking procedures. We then proceed to describe a Federated Search infrastructure to be implemented on top of these existing systems. An important feature of the proposed federation infrastructure is that participants do not have to change their existing search and cataloging systems. Communication within the federation is performed via additional plugins, which can be implemented by the participants, provided by search engine vendors or by a third party. When participating in the federation, all documents (both full-text and metadata) stay at the provider side, no library document / metadata exchange is necessary. The integration of collections is based on a common protocol, SDARTS, to be supported by every member of search federation. SDARTS is a hybrid of the two protocols SDLIP and STARTS. SDLIP was developed by Stanford University, the University of California at Berkeley, the California Digital Library, and others. STARTS protocol was designed in the Digital Library project at Stanford University and based on feedback from several search engines vendors. Additional advantages can be gained by agreeing on a common document schema, as proposed by the Vascoda initiative, though this is not a precondition for Federated Search.