{'fraction_non_alphanumeric': 0.025084238113066266, 'fraction_numerical': 0.0007487832272557095, 'mean_word_length': 5.597530864197531, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Learning the Structure of Task-Oriented Conversations from the Corpus of In-Domain Dialogs

Acquiring domain-specific knowledge necessary for creating a dialog system in a new task-oriented domain is a time consuming task that requires domain expertise. This dissertation explores the feasibility of using a machine learning approach to infer the required domain-specific information automatically from in-domain conversations. In order to achieve this goal, two problems need to be addressed: 1) creating a dialog representation that is suitable for representing the required domain-specific information, and 2) developing a machine learning approach that uses this representation to capture information from a corpus of in-domain conversations. In order to solve the first problem, I propose a form-based dialog structure representation incorporating a three-level structure of task, sub-task, and concept. These components are observable in human dialogs. In terms of representation, tasks and subtasks are represented by forms while concepts are slots in a form. The notion of form is generalized as a repository of related pieces of information so that it can be applied to various types of task-oriented domains. Dialog structure analysis and an annotation experiment are used to demonstrate that the form-based representation has all the required properties: sufficiency, generality, and learnability. The proposed representation is applied to six disparate task-oriented domains (air travel planning, bus schedule inquiry, map reading, UAV flight simulation, meeting, and tutoring). While the formbased approach shows some limitations, it is sufficient to model important phenomena in dissimilar types of task-oriented dialogs, and thus has both sufficiency and generality. The annotation experiment shows that the form-based dialog structure representation can be applied reliably by novice annotators which implies that the representation is unambiguous and learnable. For the second problem, inferring the form-based dialog structure representation from a corpus of in-domain conversations, I divide this dialog structure acquisition problem into two sub-problems, concept identification and form identification, to make the problem tractable. In order to identify a set of domain concepts, two unsupervised concept clustering approaches are investigated: statistical-based clustering and knowledge-based clustering. For most statistical-based clustering algorithms, we are able to find automatic stopping criteria that yield close to optimal results. The statistical-based approaches, which utilize word co-occurrence statistics such as mutual information and