{'fraction_non_alphanumeric': 0.04095928543024074, 'fraction_numerical': 0.009452142791594016, 'mean_word_length': 4.528834770843903, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Wizard of Search Engine: Access to Information Through Conversations with Search Engines

Conversational information seeking (CIS) is playing an increasingly important role in connecting people to information. Due to the lack of suitable resource, previous studies on CIS are limited to the study of theoretical/conceptual frameworks, laboratory-based user studies, or a particular aspect of CIS (e.g., asking clarifying questions). In this work, we make efforts to facilitate research on CIS from three aspects. (1) We formulate a pipeline for CIS with six sub-tasks: intent detection (ID), keyphrase extraction (KE), action prediction (AP), query selection (QS), passage selection (PS), and response generation (RG). (2) We release a benchmark dataset, called wizard of search engine (WISE), which allows for comprehensive and in-depth research on all aspects of CIS. (3) We design a neural architecture capable of training and evaluating both jointly and separately on the six sub-tasks, and devise a pre-train/fine-tune learning scheme, that can reduce the requirements of WISE in scale by making full use of available data. We report some useful characteristics of CIS based on statistics of WISE. We also show that our best performing model variant isable to achieve effective CIS as indicated by several metrics. We release the dataset, the code, as well as the evaluation scripts to facilitate future research by measuring further improvements in this important research direction.


INTRODUCTION
Search engines have become the dominant way for people around the world to interact with information. Today, traditional search engines (TSEs) have settled on a query-SERP (Search Engine Result Page) paradigm, where a searcher issues a query in the form of keywords to express their information need, and the search engine responds with a SERP containing a ranked list of snippets. Even though this has become the dominant paradigm for interactions with search engines, it is confined to clear navigational or transactional intents where the user wants to reach a particular site [7], or relatively simple informational intents, where the user's questions can be answered with short text spans [8]. However, it is not effective when the user has complex and composite intents, especially when their information need is unclear and/or exploratory [43].
Conversational information seeking (CIS) has emerged as a new paradigm for interactions with search engines [3,11,47]. In contrast to the query-SERP paradigm, CIS allows users to express their information need by directly conducting conversations with search engines. In this way, CIS systems can better capture a user's intent by taking advantage of the flexibility of mixed-initiative interactions, and provide useful information more directly by returning human-like responses. CIS is increasingly attracting attention. Currently, research on CIS is mostly focused on the following aspects: asking clarifying questions [2,48], conversational search [13,41], conversational question answering [29,31], and conversational recommendation [9,34]. A complete CIS system should be a mixture of these aspects and go far beyond these [16]. Other aspects, e.g., chitchat [45], user intent understanding [27], feedback analysis [19], conversation management, and so on [38], are still not well studied in the context of CIS scenario. Since no complete CIS models capable of handling all these aspects have been developed so far, there is a lack of comprehensive analysis on the performance of those aspects when achieved and/or evaluated simultaneously. This is largely because there is no practical formalism or suitable resource for CIS. As a result, related research is restricted to user studies [36,42] and/or theoretical/conceptual CIS frameworks [4,28].
We contribute to CIS by pursuing three goals in this work: a pipeline, a dataset, and a model. Let us expand on each of these.
(1) We formulate a pipeline for CIS consisting of six sub-tasks: intent detection (ID), keyphrase extraction (KE), action prediction (AP), query selection (QS), passage selection (PS), and response generation (RG), as shown in Figure 1. ID identifies the general intent behind the current user utterance, e.g., greetings ("chitchat") or revealing information need ("reveal"). Then, KE extracts keyphrases from previous conversations to represent the fine-grained information need behind the current user utterance (if any), which can be used to retrieve candidate queries (i.e., suggested queries, related queries) and candidate documents/passages. AP is used for conversation management as it decides the system action at each turn. QS and PS select the relevant queries and passages from which RG can distill useful information (if any) to generate conversational responses. We make a close connection to TSEs so as to retain its characteristics as much as possible (e.g., candidate queries and passage retrieval) while introducing new characteristics of CIS (e.g., conversation management, conversational response generation). (2) We build a dataset of human-human CIS conversations in a wizard-of-oz fashion, called wizard of search engine (WISE), where two workers play the role of seeker and intermediary, respectively. We first collect search sessions from a commercial search engine, which cover a diverse range of information needs. Then, we write guidelines for the seekers by inferring the intents behind the keyword queries in the search sessions, e.g., "[anecdotes, anecdotes in the world, Tolstoy, Tolstoy Wikipedia] → You want to find some materials about anecdotes. Describe the features of anecdotes you are interested in. Discover the introduction of the anecdotes and ask for links. " The seekers can better understand the information need when provided with guidelines than keyword queries from search sessions. We make sure the guidelines cover diverse search scenarios, e.g., when the users have specific, unclear, or multiple intents. Finally, we build a conversational interface for the workers to conduct conversations. The interface for the intermediaries is backed by a commercial search engine. We record all the operations the workers take, which can be used for supervised learning or direct evaluation for all six sub-tasks. (3) We propose a modularized end-to-end neural architecture, where each module corresponds to one of the six sub-tasks listed above. Thus, we have a natural way to train and evaluate the model, Table 1: Summary of main notation used in the paper.
User utterance at turn . represents the token at position . System utterance at turn .
represents the set of candidate passages at turn . A query ∈ Q.
represents the set of candidate queries at turn . A user intent ∈ I. A system action ∈ A. A keyphrase.
is the set of keyphrases for : + .
ℎ A hidden representation ℎ ∈ . is a sequence.
both jointly and separately, by either forwarding the ground truth or the prediction to downstream modules. We also devise a pre-train/fine-tune learning scheme to make full use of available data and reduce the requirements of supervised learning. We use four related tasks/datasets to guarantee that all the modules can be pre-trained. We conduct analyses to determine several useful characteristics of CIS as reflected in the WISE dataset. We also carry out experiments to analyze the performance of different model variants on the six sub-tasks, as well as the effectiveness of the pre-training tasks. We show that our best performing model variant is able to achieve effective CIS. To facilitate experimental reproducibility, we share the dataset, code, and evaluation scripts. Table 1 summarizes the notation used in the paper.

Task 1: Intent detection (ID)
The goal of this task is to identify the general intent of each user utterance. There have been several similar tasks [27]. However, the proposed user intent taxonomies are either too general or not well mapped to user behavior in TSEs. Considering previous work [4,28] while making a connection to TSEs, we define a new user intent taxonomy, as shown in Table 2. Most intents can map to the corresponding user behavior or operation in TSEs, except for "chitchat" and "request-rephrase", because TSEs do not have conversations and the outputs are always organized as SERPs. The algorithmic problem of ID in CIS is defined as learning a mapping: { : , } → . That is, given the context : and the current user utterance , the goal is to predict the user intent of for the utterance .

Task 2: Keyphrase extraction (KE)
Conversations usually contain more content than necessary in order to ensure the integrity of grammar and semantics, etc. Ellipsis and anaphora are common in conversations, so the current user utterance itself is usually not semantically complete. This is especially true as conversations go on and when the users are not familiar with the topic, or unclear about the information goals themselves. It is challenging to fetch documents that are relevant to the intent in the current user utterance by simply issuing the current user utterance to the TSE. To this end, similar to [41], we define KE as a task to extract keyphrases from the current user utterance as well as the previous conversational context, which can help TSEs to retrieve relevant documents for the current user utterance. Formally, KE is defined as learning a mapping: { : , } → .

Task 3: Action prediction (AP)
To connect users to the right information with maximal utility, a good CIS system should be able to take appropriate actions at the right time, e.g., helping the users to clarify their intents whenever unclear, and providing information in the right form by tracking user expectations. AP has been well studied in task-oriented dialogue systems (TDSs) in a closed-domain setting (a.k.a. dialogue policy) [26]. However, AP is not yet well addressed in CIS. Although there have been some investigations about what system actions are necessary in CIS, to the best of our knowledge, it has not yet been put into practice [4,28]. By recalling the operations in TSEs, we introduce a system action taxonomy in Table 3. The AP task is formulated as learning a mapping: { : , , , } → . Note that this task also takes the candidate queries and candidate passages fetched from TSEs as input, as they will influence the action the CIS system should take. For example, to decide whether "no-answer" is the action to be taken, the CIS system must check whether the answer is available in and/or .

Task 4: Query selection (QS)
In TSEs, a list of suggested or related queries is usually presented to users to help them clarify intents or find similar intents. In CIS, we cannot always simply list the suggested or related queries to users, as this is not natural in conversations. Previous studies cast this problem as a problem of asking clarifying questions, and model it as either selecting a question from a question pool prepared in advance [2] or generating a clarifying question for a given keyword query [48]. However, neither option fits in our formulation. Besides, how to recommend similar search intents rather than asking clarifying questions is neglected. We introduce the QS task as a step in the CIS pipeline. Depending on the predicted actions from the upstream AP task, QS aims to select queries fetched from TSEs that can be used by the downstream task to generate clarifying questions or recommend similar search intents. Formally, QS is defined as learning a mapping: { : , , , ( , )} → , where is set of suggested or related queries fetched from TSEs with the keyphrases from KE as queries, and and are the user intent and system action from ID and AP, respectively.

Task 5: Passage selection (PS)
Passage or document ranking has long been studied in TSEs [46]. Recently, this has also been investigated in a conversational context [13]. The situation is a bit different for PS in CIS where the context is not only about the proactive questions from users but also the clarifying questions or recommendations from the systems. And the goal is not always to find the relevant passages for the user questions but also non-relevant passages that contain related topics in order to do clarification or recommendation sometimes [43]. Given the same input as in QS, the PS task is to select passages based on which to generate system responses by the downstream task: where is a set of passages fetched from a TSE with the same keyphrases from KE as queries.

Task 6: Response generation (RG)
RG is a fundamental component for many NLP tasks, e.g., chitchat [30], TDS [25], conversational question answering [6]. The goals or formulations of RG vary across tasks. The role of RG in our formulation of CIS is to translate system actions and predictions of the above tasks into natural language responses. For example, if the system action is "clarify", the RG part needs to generate a clarifying question by referring to the selected queries. In this work, we only focus on free text responses. The formulation of RG is defined as a learning problem: { : , , , , ( , , , )} → .

DATASET
The WISE dataset is built based on the following setting: two participants engage in a CIS conversation, one plays the role of a searcher while the other plays the role of a knowledgeable expert (referred to as wizard). At each stage of the conversation, the searcher talks to the wizard freely. Their goal is to follow the general instruction about a chosen search intent, go into depth about it, and try to get all information that has been mentioned in the instruction. The wizard helps the searcher to find the information by interacting with a search engine. Their goal is to help the searcher (1) clarify their search intents whenever unclear, (2) answer the searcher's questions by finding and summarizing the relevant information in search results, and (3) recommend information that has not been asked but is related to the searcher's search intents or other related topics the searcher might be interested in. There are two steps in building the WISE dataset, as shown in Figure 2: collecting search intents, and collecting CIS conversations.

Collecting search intents
First, we collect a set of 1,196 search intents from a search log of a commercial search engine. Each search intent is based on a specific search session. For a given search session, e.g., "[anecdotes,  open Give an answer to an open-ended question, or one with unconstrained depth. The response is objective but may be different depending on the perspectives.
One of the reasons of the earthquake is that. . . (answer-open) answer-form free-text Answer the user intent by providing information in the right form or when being asked to answer in a particular form.  anecdotes in the world, Tolstoy, Tolstoy Wikipedia, Tolstoy movie]", the workers are asked to infer the search intent behind it using their imagination, and write down a description in one of the following forms: (1) specific intent, e.g., "You are interested in anecdotes about famous writers in the world like Tolstoy. Try to learn about his/her basic information, life story and representative works, and so on. " (2) exploratory intent, e.g., "You want to find some material about anecdotes. Describe the features of anecdotes you are interested in. Try to learn about the introduction of the anecdotes and ask for links. " (3) multiple intents, e.g., "(1) or (2) + You also want to find a movie that is based on his/her works. Ask for information about the director, plot, links and so on. "

Collecting CIS conversations
The conversation flows as follows. (1) The searcher randomly picks a search intent and starts by asking questions directly or by uttering a greeting. When sending a message, the searcher is required to select a label from Table 2 that can best describe the intent of the message.
(2) Whenever the wizard receives the message from the searcher, s/he needs to extract keyphrases from the conversation history that are used to fetch results from a search engine. Then s/he has to select a label from Table 3 that reflects the action s/he is going to take. After that, s/he needs to select the relevant queries and/or documents, based on which to formulate the responses.
(3) The conversation repeats until the searcher ends the chat (after a minimum of 7 turns each). For each turn, both the searcher and wizard can send multiple messages at once by clicking the "send another message" option. After collecting data in this way, the goal is to then replace the wizard with a learned CIS system that will speak to a human searcher instead.

Measures/Disclaimers for data quality
3.3.1 Measures. We took several measures to guarantee the data quality. First, we provided detailed documentation on the task, the definition of the labels, the steps involved in creating the data, and the instruction of using the interfaces. Second, we recorded a video to demonstrate the whole process and enhance the considerations to take into account. Third, we also maintained a list of positive and negative examples from us as well as the workers as a reference for the other workers. Fourth, after obtaining the data, we manually went over it and corrected the following issues: grammatical issues, mis-labels, long responses without summarization.

Disclaimers.
In all the provided instruction materials, we described the purpose of this data construction effort and pointed out that the data will only be used for research. We did not record any information about the workers and warned the workers not to divulge any of their private information during conversations. We filtered out all adult content and/or offensive content that might raise ethical issues when collecting the search intents, and asked the workers to be careful about such content in conversations too.

Statistics
It took 24 workers 3 months to build the dataset.

METHODOLOGY
We propose a model to replace the wizard in data construction and devise a pre-train/fine-tune learning scheme to train the model.

Model
As shown in Figure 3, we propose an architecture that relies mainly on transformer encoders and decoders.  a ReLU activation in between [1]. Before each layer, layer normalization is applied [5]. After each layer, a residual connection is applied. For any given sequence , the transformer encoder outputs a sequence of hidden representations ∈ R × corresponding to : = TEncoder( ), where is the sequence length, is the hidden size, and is short for transformer encoder.

Transformer decoder.
The transformer decoder consists of three layers: an output-attention layer, an input-attention layer and a position-wise feed-forward layer. They all use the same implementation as in the transformer encoder. The difference is that the output-attention and input-attention layers try to identify useful information in the output sequence and input sequence, respectively. For the output sequence, a look-ahead mask is used to mask the future positions so as to prevent information from future positions from affecting current positions [39]. Again, before and after each layer, a layer normalization and a residual connection are applied, respectively. For any given output and input sequences and , the transformer decoder outputs a sequence of hidden representations ∈ R × corresponding to and the input-attention weights ∈ R × corresponding to : , = TDecoder( , ), where and are the output and input sequence lengths, respectively. TDecoder is short for transformer decoder.  . To do so, the same stack of transformer encoders as for ID is used to get the hidden representations for KE based on the representations from ID: 2 = TEncoder( 1 ). A linear classifier with sigmoid is used to predict whether each token belongs to a keyphrase or not.

Task 3: Action prediction (AP)
. To predict the system action, we also need to consider the query and passage candidates and . For each or , we first input it into a token and positional embedding layer to get or . We then concatenate it with the hidden representations from KE and fuse the context and query/passage information with the same stack of transformer encoders as for ID: 3 = TEncoder([ 2 , ]) and 3 = TEncoder([ 2 , ]). Note that another special token "[T3]" is used (see Figure 3). A hidden representation corresponding to "[T3]" is obtained for each or . We combine them with a max pooling to get a single hidden representation, on top of which a linear classifier with softmax is applied to do AP.

Task 4: Query selection (QS).
For QS, a similar architecture as AP is adopted. The same stack of transformer encoders is used to get hidden representations for QS based on the hidden representations 3 from AP: 4 = TEncoder( 3 ). We get the hidden representation corresponding to "[T4]", on top of which a binary classifier with sigmoid is used to predict whether a query should be selected or not. The hidden representation corresponding to "[T5]" is used to predict whether a passage should be selected or not using a binary classifier with sigmoid.

Task 6: Response generation (RG).
We use a stack of transformer decoders to do RG. The output sequence in the transformer decoder is the ground truth response during training or the generated responseˆduring inference. Note that we put the system action at the beginning to indicate the type of the response. We get the fused hidden representations by taking the context, query, and passage hidden representations from KE, QS, and PS as the input sequence in the transformer decoder, respectively. Specifically, 6 , 6 = TDecoder( , 2 ); 6 , 6 = TDecoder( , 4 ); 6 , 6 = TDecoder( , 5 ). 6 , 6 and 6 are fused again with max pooling. Then a linear classifier with softmax is applied on top to predict the probability of the token at each time step ( ). We also extend the pointer mechanism [40] to generate a token by copying a token in the context, query or passage [33]. Pointer mechanisms are commonly used in natural language generation tasks. The copying probability is modeled as ( ) = max( 6 ( ), ( ) 6 ( ), ( ) 6 ( )), where 6 ( ) is the sum of attention weights for in the context. A similar definition applies to 6 ( ) and 6 ( )). ( ) and ( ) are the selection probability from QS and PS, respectively. The final probability ( ) is a linear combination of ( ) and ( ).

Learning
We use binary cross-entropy loss for KE, QS and PS and crossentropy loss for ID, AP and RG. To solve complex tasks with a deep model, a large volume of data is needed. We get around this problem by first pre-training the model on four datasets from related tasks, as shown in Table 5. WebQA is a dataset for single-turn question answering. For each sample, it provides a simple question in natural language, a set of candidate passages, and an entity answer. DuReader is for single-turn machine reading comprehension. The questions are mostly complex and the answers are summaries of
Implementation details. For a fair comparison, we implement all models used in our experiments based on the same code framework. We set the word embedding size and hidden size to 512. We use the same vocabulary for all methods and its size is 21,129. The learning rate was increased linearly from zero to 2.5 × 10 −4 in the first 6,000 steps and then annealed to 0 by using a cosine schedule. We use gradient clipping with a maximum gradient norm of 1. We use the Adam optimizer ( = 0.001, 1 = 0.9, 2 = 0.999, and = 10 −8 ). We use 4 transformer layers for the encoder and 2 for the decoder, where the number of attention heads is 8. In WISE, the parameters of the transformer encoder, transformer decoder, and word embedding for each component of our model are shared. We train all models for 50 epochs, evaluate on a validation set for the best model in terms of BLEU-1 in RG task.

RESULT AND ANALYSIS 6.1 Performance analysis
We report the results of all methods on WISE; see Table 6. We consider three settings: (1) With pretraining (i.e., WISE in Table 6).
(2) Without pretraining (i.e., WISE-pretrain in Table 6). (3) With pretraining and using the ground truth of ID, KE, AP, QS and PS (i.e., WISE+GT in Table 6). We also report the performance of WISE on the test (unseen) and test (seen) datasets, and on different actions. The results are shown in Table 7 and 8.
From Table 6, there are two main observations. First, WISE+GT achieves the best performance in terms of all key metrics. As we can see in Table 6, WISE+GT achieves 20.4% and 24.2% in terms of BLEU and ROUGE for RG, outperforming the strongest model WISE by 5.8% and 7.3%, respectively. This shows that the human labels of ID, KE, AP, QS, PS do benefit RG.
Second, WISE achieves a higher performance than WISE-pretrain in terms of most metrics, indicating that the pretrain phrase is of great benefit for the performance of WISE. For example, the BLEU and ROUGE score of WISE for RG are 13.6% and 15.9%, much higher than those of WISE-pretrain, respectively. The reason is that the model can leverage external knowledge in the pretrain phrase, which boosts the final performance.
From Table 8, we obtain three main findings. First, for the action 'Clarify', the best performance in terms of KE, AP and QS is achieved w.r. Third, the performance of PS has a high correlation with the final RG task w.r.t. 'Answer', while the performance of QS does not seem to be correlated w.r.t. 'Clarify'. As we can see in Table 8, an F1 score of 23.3% in PS and BLEU score of 12.6% in terms of RG is achieved w.r.t. 'Answer opinion', and a higher F1 score (30.0%) in terms of PS and a higher BLEU score (16.4%) in terms of RG w.r.t. 'Answer fact'. For response generation w.r.t. 'Answer', large portions of the output can be copied from passages, and thus the performance PS is crucial to the final RG performance. The F1 score of WISE is 10.9% in QS w.r.t. 'Clarify yes-no', which is lower than 29.6% w.r.t. 'Clarify choice'; the performance in RG is better, e.g., 28% vs. 20%. It is more likely that WISE has to copy from multiple queries for 'Clarify choice' and from a single query for 'Clarify yes-no', leading to performance differences for QS and RG, w.r.t. 'Clarify'.
We split the test dataset based on the occurrence of conversational background in the training dataset: (test (unseen) means no occurrence, and test (seen) means occurrence); the results are shown in Table 7. Generally, the performances of WISE on all tasks is best on the test (seen) dataset, and worst on the test (unseen) dataset. For example, it achieves 29.6% in terms of F1 for PS in test (seen) dataset, which is higher than 24.9% in test dataset and 20.5% in test (unseen) dataset. WISE has been trained on the same conversational background in the test (seen) condition, and has never seen the conversational background in the test (unseen) condition.

Effects of subtasks
We analyze the effects of different tasks in WISE, w.r.t. ID, KE, AP, QS and PS, and the results are shown in Table 9. We remove each task, resulting in five settings; '-xx' means WISE without module 'xx'. Generally, removing any module will result in a drop in performance for RG on all metrics, indicating that all parts are helpful to WISE. Without QS, RG drops sharply on all metrics. Specially, it

Effects of pretraining
To analyze the effects of different pretrain datasets, we conduct an ablation study (Table 10). WISE uses four external datasets for pretraining; each time we remove one, we use the others for pretraining. Hence, we have four conditions. In Table 10, '-xx' means pretraining WISE using datasets other than 'xx'.
The results show that all datasets are helpful for pretraining; removing any dataset results in a performance drop for RG. The results drop in terms of all metrics when pretraining without the DuReader dataset. The results drop by 5.0% and 5.9% in terms of BLEU and ROUGE for RG. The drop after removing pretraining with DuReader is far larger than after removing any of the other individual pretraining dataset, indicating the importance of DuReader for pretraining, as shown in Table 5. Interestingly, sometimes removing a dataset will improve the performance of ID and AP. For example, -duconv achieves 35.8% in terms of F1 in ID, higher than the 34.1% for WISE. ID and AP predict labels among small sets, which are relatively easy compared to the other tasks.

RELATED WORK
User studies. To determine whether CIS is needed and what it should look like, researchers have conducted different user studies. Vtyurina et al. [42] ask 21 participants to solve 3 information seeking tasks by conversing with three agents: an existing commercial system, a human expert, and a perceived experimental automatic system, backed by a human "wizard behind the curtain. " They conclude that people do not have biases against CIS systems, as long as their performance is acceptable. Trippas et al. [37] conduct a laboratory-based observational study, where pairs of people perform search tasks communicating verbally. They conclude that CIS is more complex and interactive than traditional search. Based on their conclusions, we can confirm that CIS is the right direction but there is still a long way to go.
To address this gap, there are various efforts to build datasets for CIS. Dalton et al. [12] release the CAsT dataset, which aims to establish a concrete and standard collection of data to make CIS systems directly comparable. Ren et al. [31] provide natural  language responses by summarizing relevant information in paragraphs based on CAsT. Aliannejadi et al. [2] explore how to ask clarifying questions, which is important in order to achieve mixedinitiative CIS. They propose an offline evaluation methodology for the task and collect the dataset ClariQ through crowdsourcing. The above datasets only cover some aspects of CIS.
To better understand CIS conversations, Thomas et al. [35] release the MISC dataset, which is built by recording pairs of a seeker and intermediary collaborating on CIS. The work most closely related to ours is by Trippas et al. [38]. They create the CSCData dataset for spoken conversational search (very close to the CIS setting), define a labelling set identifying the intents and actions of seekers and intermediaries, and conduct a deep analysis. The MISC and CSCData datasets only have 666 and 1,044 utterances, which results in a low coverage of search intents and answer types. No supporting documents are available and the responses are in very verbose spoken language. Importantly, it is hard for the defined labels to make a direct connection to TSEs.
Frameworks. Several recent frameworks address a specific aspect of CIS, e.g., asking clarifying questions [48], generating responses for natural language questions [23]. There are also attempts to design a complete CIS that could work in practice. Radlinski and Craswell [28] consider the question of what properties would be desirable for a CIS system so that the system enables users to answer a variety of information need in a natural and efficient manner. They present a theoretical framework of information interaction in a chat setting for CIS, which gives guidelines for designing a practical CIS system. Azzopardi et al. [4] outline the actions and intents of users and systems explaining how these actions enable users to explore the search space and resolve their information need. Their work provides a conceptualization of the CIS process and a framework for the development of practical CIS systems.
To sum up, the work listed above either focuses on a particular aspect or studies the modeling of CIS theoretically.

CONCLUSION AND FUTURE WORK
We have proposed a pipeline and a modular end-to-end neural architecture for conversational information seeking (CIS), which models CIS as six sub-tasks to improve the performance on the response generation (RG) task. We also collected a WISE dataset in a wizard-of-oz fashion, which is more suitable and challenging for comprehensive and in-depth research on all aspects of CIS. Extensive experiments on the WISE dataset show that the proposed WISE model can achieve state-of-the-art performance and each proposed module of WISE contributes to the final response generation.
Although we have proposed a standard dataset for CIS, we have a long way to go to improve the performance on each subtask, and to consider more factors, e.g., the information timeliness [15]. A potential direction is to leverage large-scale unlabeled datasets on related tasks like dialogue and information retrieval.

REPRODUCIBILITY
The code and data used to produce the results in this paper are available at https://github.com/PengjieRen/CaSE_WISE.