{'fraction_non_alphanumeric': 0.03003552026744672, 'fraction_numerical': 0.009454659423318012, 'mean_word_length': 5.281167979002625, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 4, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A Conceptual Framework for Implicit Evaluation of Conversational Search Interfaces

Conversational search (CS) has recently become a significant focus of the information retrieval (IR) research community. Multiple studies have been conducted which explore the concept of conversational search. Understanding and advancing research in CS requires careful and detailed evaluation. Existing CS studies have been limited to evaluation based on simple user feedback on task completion. We propose a CS evaluation framework which includes multiple dimensions: search experience, knowledge gain, software usability, cognitive load and user experience, based on studies of conversational systems and IR. We introduce these evaluation criteria and propose their use in a framework for the evaluation of CS systems.


Introduction
Recent progress in artificial intelligence has brought tremendous advances in conversational systems and information retrieval (IR). This has led to increasing interest in Conversational Search (CS) using conversational engagement to complete IR tasks [30]. CS presents opportunities to support users in their search activities to improve the effectiveness and efficiency of information seeking, while reducing their cognitive load. A number of studies have been conducted to examine the concept of the CS [20,30]. We believe that greater insight into the processes and potential of CS can be achieved using a detailed evaluation, and that this can help in advancing and understanding the exploitation of the paradigm of CS. These insights will help in enhancing proposed models and theories of CS. This paper overviews 1 the current methods and techniques used in the evaluation of conversational systems in different dimensions, and use them to define a framework for the definition and utilization of evaluation metrics for CS.

Background
This section introduces existing work examining the evaluation of interactive IR, conversational systems and CS.

Evaluation of Conversational Systems
Conversational engagement is currently being investigated as the mode of engagement for many human-machine applications. Research in conversational systems has proposed 6 dimensions for the evaluation of conversational agents [31,43].
-Extensive Capabilities: Conversational systems should provide an errorfree environment to the user [29,31,43]. This should include spell checking and auto-correction to support error free expression by the user of statements and questions. In addition, conversational systems should use an appropriate combination of multimedia and text content [10,45]. -User Interaction and Engagement:This measure contains the following parameters: capability of initiating conversations, maintaining conversational engagement, identifying and distinguish target users, etc. [42]. -Response speed: The response speed of conversational agents should be sufficiently fast to prevent user frustration [31,37,43].
-Functionality: The overall capabilities of the system as measured qualitatively by users based on multiple variables such as the richness of media supported, navigation tools provided to support users, multi-modality of engagement, etc [29,36]. -Interoperability: This defines the ability of a conversational system to exchange and make use of information. A standard conversational system enables the user to engage with multiple media channels [43]. -Scalability: This is a quantitative measure of the scope of the system to support multiple users. For example, the number of users supported by the conversational system at the same time, types of server that can accommodate the conversational system, database size, etc [43].
A well known framework for evaluation of conversational systems is the Paradise framework [44]. This includes measure such as: task success, conversation efficiency (task duration, dialogue turns), conversation quality (response accuracy), and user satisfaction (ease of the task, user behaviour). However, Paradise is of limited value for the evaluation of of conversational search systems, since it does not include important factors such as cognitive load and the knowledge gain during the search process. Moreover, Paradise focuses on a goal-oriented agent, which is different from a search-oriented task for which the requirements can change as the user progresses through the search process. Hence, Paradise is not suitable for evaluation of conversational search systems.

Evaluation of Conversational Search
In recent years, a number of studies have been conducted on CS systems and interfaces. There studies can be broken down into four different approaches: using existing conversational agents [23,25], using human experts [38,39], human covertly taking the role of automated agent (Wizard-of-Oz approach), [3,6,28], and using rule-based or machine learning conversational interfaces [19,21]. Evaluation in most of this work is limited to NASA TASK Load [12] or SUS (Usability) [5] or both [4,9]. Some of these studies have also investigated sentiment [9] in the user response to examine the relationship between the user's mood and task success.
There is also active research exploring the use of evaluation benchmarks for CS. Most notable is the TREC CAsT track [2,7]. An alternative interpretation of CS is examined in the FIRE RCD track [18]. These tasks have examined query interpretation and response in the context of conversational engagement and query extraction from conversations respectively. In both cases, evaluation is largely limited to traditional approaches used for IR tasks.
Evaluation of interfaces in CS is a highly complex topic involving multiple dimensions including the user's background knowledge of the search topic, their familiarity with the conversational agent, etc. In this paper, we propose a framework for the evaluation of CS interfaces using five dimensions: user search experience [20], knowledge gain [46], cognitive and physical load [12], usability of the interface software [27] and user experience [13].

Framework for the Implicit Evaluation of Conversational Search Interfaces
In this section we introduce our framework for the evaluation of CS which focuses on multiple dimensions relating to the user. Most CS studies so far reported have focused on user search experience of the task or the usability of CS systems. This has provided feedback focused on user search experience. In our framework, evaluation is based on five factors responsible for the needs of CS outlined in the next section. An approach of this sort is also advocated in the summary report from the Dagstuhl Seminar on CS [1].

Essential Factors for Conversational Search
We identify the following essential factors for the evaluation of CS interfaces.
1. Cognitive Load: Conventional search can impose a significant cognitive load on the searcher [17]. An important factor in the evaluation of conversational systems is measurement of the cognitive load experienced by users while using the system. 2. Cognitive Engagement: It has been observed that users get frustrated if they find it difficult to search about their topic of interest to satisfy their information need. Frustration can reduce the user's engagement with a search system and their associated effort to locate relevant information. 3. Search as Learning: Learning while searching is an integral part of the information seeking process. In our current study, we propose a metric which breaks this down into three phases [20,40], as shown in Table 1. It has been observed in conventional search that high cognitive load and lower cognitive engagement impacts on user learning during the search process [11]. 4. Knowledge Gain: Satisfaction of the user's information need is directly related to their knowledge gain about the search topic. Knowledge gain can be measured based on recall of new facts gained after the completion of the search process [46]. 5. User Experience (UX): Another important aspect that needs to be considered for evaluation of CS systems is UX. User experience is generally classified into two aspects: pragmatic and hedonic, which can be further divided into six components: attractiveness, perspicuity, efficiency, dependability, stimulation and novelty [13]. These factors all provide measures of user ease of use and the dependability of a conversational system [13]. 6. Software Usability: CS studies generally do not explore the dimensions of software usability. However, it is important to understand the challenges and opportunities of conversational systems on the basis of software requirements analysis. This allows a system to be evaluated based on real life deployment and to identify areas for improvement. Lower effectiveness and efficiency of a software system can increase cognitive load, reduce engagement and act as a barrier in the process of learning while searching.

Designing our Conceptual Framework
In our work on CS, we have conducted multiple studies of CS and the use of associated conversational agents. In conjunction with this, we have also conducted four user studies on conventional IR systems, a commercial conversational system (Alexa Echo Show) and conversational search interfaces for complex search tasks. In these studies, we have examined user behaviour and user expectations with respect to CS. Based on our investigations [2,[17][18][19][20], we propose an evaluation framework for CS interfaces. The framework is divided into two segments: Exploration and Contentment, as shown in Figure 1. This section outlines the combination of standard questionnaires of multiple dimensions which form our proposed Implicit Evaluation for Conversational Search Interface (IECSI). The details are as follows: Explore Segment: This segment focuses on exploring and experiencing CS interfaces, and is classified into three components: Software Usability, User Experience, and Cognitive Load.
1. Software Usability: Usability is an important consideration for the evaluation of interactive software. the IBM Computer Usability Satisfaction Questionnaires enables psychometric evaluation from the perspective of the user, and is known as the Post-Study System Usability Questionnaire (PSSUQ) [27]. The PSSUQ includes four dimensions: overall satisfaction score (OVER-ALL), system usefulness (SYSUSE), information quality (INFOQUAL) and interface quality (INTERQUAL), which includes sixteen parameters. 2. User Experience: UX is measured using a questionnaire for interactive product known as the User Experience Questionnaire (UEQ-S) [13,26,35]. This questionnaire also enables us to analyse and interpret outcomes by comparing against a benchmark dataset of outcomes for other interactive products. This questionnaire also provides us with the opportunity to compare interactive products with each other. UEQ-S contains two meta quality dimensions: pragmatic and hedonic. Each dimension contains four different parameters as shown in the Table 2. Pragmatic quality explores the usage experience of a conversational search system. Hedonic quality explores the pleasantness of using the system. 3. Cognitive Load: An important consideration in the evaluation of CS interfaces is their impact on the user's cogntive load during the search process. To measure the user's workload, the NASA Ames Research Centre proposed the NASA Task Load Index [12]. This is a multi-dimensional rating procedure which provides a measurement of the overall workload during a process or event. This workload is classified into six subscales: mental, physical, temporal, own performance, effort and frustration. Out of these six dimensions, three are related to the demand imposed on the subject due to the task (mental, physical and temporal) and the remaining three to the interaction of the subject with the system (effort, frustration and performance). This implicit evaluation enables us to examine the cognitive load and cognitive engagement of the user while using a system.  Table 3. Summary Comparison Metric [46] 1. Search as Learning: As discussed in Section 3.1, it is important to observe whether a CS system supports the user effectively in their engagement with the search system, and enables the user's knowledge gain arising from the search process. To better understand this process, we decided to separately measure the factors of user interaction and modification of their mental knowledge structures. We developed a questionnaire [20], as shown in Table 1, to observe user interaction behaviour, inspired by Vakkeri's model of search as learning [40,41]. 2. Knowledge Gain: To measure the knowledge gain, the user is required to write a pre-search summary and a post-search summary relating to the search topic. This summary is manually evaluated by an independent assessor on three sub dimensions: Quality of Facts (Dqual), Interpretation (Dintrp) and Critiques (Dcrit), as shown in Table 3 [46]. This difference between these summaries indicates knowledge gain from the search process.

Developing the Evaluation Process
The overall evaluation process is shown in the Figure 2. The user completes a pre-search questionnaire and then a post-search questionnaire. The questionnaire is based on the metric discussed in Section 3.2, and as shown in Figure 3. To maintain uniformity, subjects rate each parameter on a 7-point Likert scale [16],  [27]. Pre-search and post-search questionnaire scores for each dimension of evaluation can be compared using their standard benchmarks. This not only provides an estimated evaluation of its effectiveness, but also provides an opportunity to explore the conversational interface with the standard system benchmarks. Furthermore, this allows us to understand user expectation in general in all dimensions. This provides empirical measurability of the CS interface in the light of the benchmark. Moreover, this can help us to understand how far from or close to, the current CS interface is to the user's expectations.
As per above the prospective, it is very important to analyse the data critically including use of statistical significance tests. If the results are significant, this can be used to develop a separate benchmark for the CS interface to assist other researchers in comparing their studies on CS interfaces.

Implementation and Analysis of the Framework
As mentioned earlier, the user is required to complete pre-search and post search questionnaires. We have developed prototype questionnaires by combining the dimensions mentioned earlier in Section 3.2 2 . The details of our pre-search and post-search questionnaires are described below.  Each question is evaluated based on a Likart score [0,7], except for the knowledge gain metric. As described, the framework is classified into two sections: exploration and contentment, as shown in Figure 3. Details are as follows: 1. Exploration : The questionnaire to investigate exploration is aligned to the user based on their search experience. As such, the conversational interface is evaluated based on the post-search questionnaire. The mean score of each question is calculated based on the number of users. Analysis is conducted using both Quantitative Analysis and Qualitative Analysis.
(a) Quantitative Analysis: This is based on the mean score of the participants in the study, statistical testing is carried out based on the population and nature of the experiment. When comparing a conventional system and a conversational system, we are able to perform dependent significant testing, since the population undertaking the experiment in both settings is the same. And, if we are comparing the mean score of the conversational interface with a standard benchmark, we can conduct independent significance testing. This statistical testing enables us to understand how systems differ. Additionally, each dimension discussed above in Section 3.2 has a standard tool for analysis. (b) Qualitative Analysis: The different dimensions are annotated based on comparison of the mean values for the study participants. A mean value between 2 and 4 represents a neutral evaluation of the corresponding scale (yellow dimension), a mean > 4 represents a positive evaluation (green dimension) and mean < 2 represents a negative evaluation (blue dimension). After comparing the mean, each question is annotated based on the dimensions. The dimensions are annotated by two independent analysts with the Kappa coefficient (Approx .85), then the dimension is counted for each section such as software usability, user experience [33,34], cognitive load. As per the dimension, the section of the interface that needs to be improved can be identified. For example, if software usability gets more red dimensions, then the interface needs to be improved with respect to software usability. 2. Contentment: The questionnaire to investigate contentment is aligned to the user's pre-search knowledge and post-search knowledge. As discussed earlier, contentment evaluation is designed to investigate user learning while searching, and their knowledge expansion arising from the search process. The analysis can again be conducted using both Quantitative Analysis and Qualitative Analysis. (a) Quantitative Analysis: Based on the mean score for the study participants of searching as learning and knowledge gain (the difference between pre-search and post-search summaries of each setting (conventional system and conversational system)) parameters, statistical testing can be applied. (b) Qualitative Analysis: Search as learning questions, as shown in Table 1, are annotated, evaluated and analyzed based on different dimensions as discussed in the previous section on Qualitative Analysis in Exploration (corresponding scale (yellow, green and red dimensions)). Pre-search and post-search summaries can be compared based on the parameters discussed in Table 3. The summary is scored against all these factors by two independent analysts with the Kappa coefficient (Approx 0.85) [24]. For each parameter, the difference between pre-search and post-search summaries is calculated. If the difference of Dqual> 1.5, Dintrp > 1 and Dcrit > 0. it is assumed that the user has increased their knowledge by more than 50%.

Concluding Remarks
The concept of conversational search (CS) remains an ongoing topic of research. A crucial part of this work is evaluation of CS. Studies of CS to date have mainly been based on user experience. This overlooks interaction with the system and changes in the user's knowledge structure. In this paper, we examine the factors that can impact on the effectiveness of a CS interface. Following our investigation, we propose an evaluation framework for CS that incorporates the evaluation methods from interactive IR and conversational systems. We believe our proposed evaluation framework for CS to be practical and applicable in real life scenarios, and will provide greater insights for understanding and advancing CS processes than is possible using the evaluation methods used in existing work on CS. We are currently working on the validation of our proposed framework within our ongoing study of CS.