{'fraction_non_alphanumeric': 0.017771883289124667, 'fraction_numerical': 0.00636604774535809, 'mean_word_length': 5.264119601328904, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Enabling Users to Visually Evaluate the Effectiveness of Different Search Queries or Engines

Abstract This paper explores how information visualization can help users evaluate the effectiveness of different query formulations or the same query submitted to multiple search engines. MetaCrystal is used to visualize the degree of overlap and similarity between the results returned by different queries or engines. It enables users to identify documents found by multiple queries or engines. Such documents tend to be more relevant and users can use their number and distribution patterns as a visual measure of the effectiveness of their search. Keywords: information visualization, search effectiveness measures, meta search. 1. Introduction When searching the Internet, users are often confronted with an overwhelming number of potentially relevant documents to sift through. It is difficult for users to determine the overall effectiveness of their search. Researchers tend to use measures, such as precision and recall , to capture the effectiveness of an information retrieval system [12]. Searching the Web is a highly interactive process and additional measures are needed to capture the richness of user interactions and their experiences of ease of use. Spink and Wilson [14] have proposed that some of these new measures need to reflect how users progress through the different stages of their information seeking process and how their interactions change their understanding of their information problem. Spink [13] employed such a user-based approach to evaluate the meta search interface Inquirus. Her study showed that users experienced some change in their information problem and information seeking stage. The results also showed that search precision did not necessarily correlate with the user-based evaluation measures or change in the search process. Another study [17] evaluated the performance of four search engines by using these user-centered criteria: relevance, efficiency, utility, user satisfaction, and connectivity. This paper explores how MetaCrystal, a set of visual tools, can help users evaluate the effectiveness of their own searches. Documents found by multiple retrieval methods are more likely to be relevant [4, 11]. MetaCrystal enables users to identify documents found by multiple queries or engines. Users tend to create short queries when searching the Internet and they rarely formulate advanced queries [15]. Eastman and Jansen [3] have shown that the use of most query operators in short Internet queries had no sig-nificant impact on the effectiveness of the search results. MetaCrystal can visualize this high degree of similarity between different formulations of simple Internet searches. It enables users to see that these related queries are not very effective in finding more relevant documents and that the relevance of the found documents can not be corroborated by these related queries. Users employ meta search engines because individual search engines only index 20% of the Internet [10] and therefore return different documents for the same query. Meta search engines address this limitation by combining the results by different engines. MetaCrystal visualizes the precise overlap between the top documents retrieved by different search engines. It makes it easy for user to identify how many and which documents have been found by more than one search engine. This paper is organized as follows: section 2 briefly reviews related visualization work. Section 3 describes the MetaCrystal toolset. Section 4 shows how the degree of similarity between different ranked lists can be visual-ized. Section 5 discusses how MetaCrystal enables users to get a visual sense of the effectiveness of their searches. 1