{'fraction_non_alphanumeric': 0.03510226644555003, 'fraction_numerical': 0.012990602542841349, 'mean_word_length': 4.957738748627881, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A Parameterized and Annotated Spoken Dialog Corpus of the CMU Let’s Go Bus Information System

Standardized corpora are the foundation for spoken language research. In this work, we introduce an annotated and standardized corpus in the Spoken Dialog Systems (SDS) domain. Data from the Let's Go Bus Information System from the Carnegie Mellon University in Pittsburgh has been formatted, parameterized and annotated with quality, emotion, and task success labels containing 347 dialogs with 9,083 system-user exchanges. A total of 46 parameters have been derived automatically and semi-automatically from Automatic Speech Recognition (ASR), Spoken Language Understanding (SLU) and Dialog Manager (DM) properties. To each spoken user utterance an emotion label from the set garbage, non-angry, slightly angry, very angry has been assigned. In addition, a manual annotation of Interaction Quality (IQ) on the exchange level has been performed with three raters achieving a Kappa value of 0.54. The IQ score expresses the quality of the interaction up to each system-user exchange on a score from 1-5. The presented corpus is intended as a standardized basis for classification and evaluation tasks regarding task success prediction, dialog quality estimation or emotion recognition to foster comparability between different approaches on these fields.


Introduction
Increasingly, data-driven techniques are employed in Spoken Dialog Systems (SDS) research with the aim of rendering SDSs more user-friendly and adaptive. As most studies rely on proprietary and non-publicly available corpora and as resources for comparisons are sparse, a transparent assessment of novel techniques is hardly possible. The Language Technology Institute (LTI) at Carnegie Mellon University (CMU) in Pittsburgh has taken the initiative to make results comparable within the framework of scientific and research activities by introducing the Spoken Dialog Challenge (Black and Eskenazi, 2009). For this purpose, both, the architecture and source code of the CMU Let's Go Bus Information system 1 as well as Let's Go interaction data collected in the field have been made publicly available. While we consider this as a cornerstone towards more transparency in research, we still felt the need for standard SDS corpora with a clearly defined scope and manageable size facilitating comparisons. Our aim in this contribution is to provide such a standardized, parameterized and welldefined corpus of manageable size that may be used in a variety of data-driven classification tasks. We propose to use this corpus for evaluating classifiers e.g., in assessing user satisfaction, and moreover, for acoustic and linguistic classification tasks, such as in emotion recognition as well as age and gender detection. The data set is based on Let's Go data from 2006 and has been preprocessed and parameterized with interaction parameters. Moreover, it has been manually annotated with interaction quality scores both, on dialog and exchange level, task completion labels and emotional states. The corpus is presented in such a manner that it may be promptly used for machine learning. 1 Let's Go delivers bus schedule information to citizens of the city of Pittsburgh. It was created at Carnegie Mellon University (CMU) and answers 40-60 calls a day (in 2006), (Raux et al., 2006).

State of the Art
Pattern classification will allow adaptiveness in future HCI. This may be achieved by introducing statistical classifiers that use learned patterns to predict interaction-related knowledge. For example, (Walker et al., 2002) introduced a classifier estimating task success in spoken dialog. The industrial corpus that was applied for this study was derived from the AT&T How May I Help You (Gorin et al., 1996) system and is consequently not publicly available. In a similar spirit (Paek and Horvitz, 2004) and (Schmitt et al., 2010b) have implemented task success prediction. Evaluation has respectively been conducted on closed corpora from Microsoft and SpeechCycle, both not available to the community. Studies dealing with modeling acoustic properties of user speech, such as emotions, age or gender categories, further frequently employ closed industrial data sets, e.g., (Metze et al., 2007), (Schmitt et al., 2010a), (Lee and Narayanan, 2005). Also studies addressing automatic evaluation of SDS, such as PARADISE-style models (Walker et al., 2000) are based on closed data sets, see also (Engelbrecht et al., 2008), (Möller, 2005) and (Rieser and Lemon, 2008).

Corpus Preprocessing
Our contribution can be summarized as follows • Formatting and Parameterization: Raw data from the log files has been transformed to a well-defined format, where each system-user exchange is represented as one logic entity. Each exchange has been parameterized with interaction parameters that quantify the interaction behavior of the user. The parameters may serve as input variables for a variety of classification tasks.
• Annotation: The corpus has been annotated with a number of labels that may serve as target variables for classification. They are the Interaction Quality (Schmitt et al., 2011), i.e., expert quality scores for each exchange and the emotional state.

Formatting and Parameterization
The raw log information has been transferred to a common structure of a system-initiative directed spoken dialog, where an exchange e comprises interaction data from system turn s and user turn u, cf. Figure  Interaction parameters were created to model each exchange as well as its immediate context and further summarize the interaction that has taken place until the current exchange. For this, we use parameters on three levels: exchange level parameters, window level parameters, and dialog level parameters. The modeling levels are depicted in Figure 2.
On the exchange level, we modeled each system-user exchange with a number of Speech Recognition (ASR), Spoken Language Understanding (SLU) and Dialog Manager (DM)-related features. These features are automatically derived from system log files. A list of all used features can be found in Table 1.
To account for the overall history of important system events, we introduced dialog level parameters by adding running tallies, percentages and mean values for certain features symbolized with the suffixes '#', '%' and 'MEAN'. Further, we consider the immediate context within the previous 3 turns of the current turn as particularly relevant for predicting target variables. Hence, derived from the basic parameters we created further window level parameters that emphasize specific user behavior prior to the classification point. They are symbolized with the prefix {#} for a number and {Mean} for the mean value. We further introduced a semi-automatically determined dialog act feature group: DAct SYSTEMDIALOGACT: one of 28 distinct dialog acts, such as greeting, offer help, ask bus, confirm departure, deliver result, etc. USERDIALO-GACT: one of 22 distinct DAs, such as confirm departure, place information, polite, reject time, request help, etc.
All presented features are calculated automatically, with exception of DAct, without manual annotation or intervention, which would make them for a real-time deployment suitable.

Annotations (Target variables)
The corpus has been annotated with a number of target variables, which are  (Schmitt et al., 2011), we annotated the corpus with Interaction Quality scores. Three raters annotated 200 dialogs (each dialog was rated by each rater) comprising 4,885 system-user exchanges. The raters were asked to annotate the quality of the interaction at each system-user exchange with the scores 5 (satisfied), 4 (slightly unsatisfied), 3 (unsatisfied), 2 (strongly unsatisfied) and 1 (extremely unsatisfied). To ensure quality, guidelines for the annotation have been developed beforehand. These can be seen in Table 3. Every dialog is initially rated with a score of 5 since in every interaction at the beginning the user can be considered as being satisfied with the dialog until the opposite eventuates. The final IQ score for an exchange is determined by creating the median of all three raters.
Emo We further introduce the negative emotional state of the user that is manually annotated by a human rater who chooses one of the labels garbage, non-angry, slightly angry, very angry for each single user turn. From all 4,832 user turns, 68.5% were non-angry, 14.3% slightly angry, 5.0% very angry and 12.2% contained garbage, i.e., non-speech events.
Task Success Each call has been annotated semiautomatically with a Task Success label, which is one of completed (187), failed due to system behaviour (15), found out that there is no solution (52), not completed (71) and partially completed (3) 2 . This was derived using a heuristic scheme, where the number of REQUEST, CONFIRMATION, and ERROR actions and the number of NO-MATCHES has been used. We will briefly describe the function of the heuristic. If at least one CONFIRMATION action has been performed, the call is labeled as completed in case of no left is the current turn a help request? OPERATORREQUEST?
is the current turn an operator request?
Dialog Manager ACTIVITY identifier of the current system action ACTIVITYTYPE one of 'question', 'announcement', 'wait for user feedback' PROMPT system prompt WPST number of words per system turn REPROMPT? is the current system turn a reprompt? CONFIRMATION? whether the current system prompt is a confirmation to elicit common ground between user and system due to low ASR confidence TURNNUMBER current turn DD dialog duration up to this point in seconds  open requests, and as partially completed otherwise 3 . If no CONFIRMATION action has been performed at all, there are three possible distinctions: First, NO-MATCHES occurred. Then, the call was labeled as failed due to system behaviour. Second, an ERROR action has been performed (e.g., destination was not covered by the system). Then, the call was labeled as found out that there is no solution. Finally, the call was labeled as not completed for all other cases.

Corpus Details
The details of the corpus are depicted in Table 2. In order to take into account the ordinal character of the IQ scores, Cohen's κ has been used with additional weights w. These weights were determined by the numerical distance d between the ratings, resulting in where k is the number of different IQ scores. By this, the penalty for small differences between the raters is not as high as for unweighted κ, which would be 1 for every mismatch. Achieving a κ value of 0.54 is a good result considering the difficulty of this task.

Download
The corpus is deployed in form of CSV files and SQL dumps and may be downloaded at http://nt. uni-ulm.de/ds-lego. We encourage to add additional labels and annotations to this data set.