{'fraction_non_alphanumeric': 0.037472224092653696, 'fraction_numerical': 0.017843916234596996, 'mean_word_length': 5.0593635250918, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
On the Need for Thoughtful Data Collection for Multi-Party Dialogue: A Survey of Available Corpora and Collection Methods

We present a comprehensive survey of available corpora for multi-party dialogue. We survey over 300 publications related to multi-party dialogue and catalogue all available corpora in a novel taxonomy. We analyze methods of data collection for multi-party dialogue corpora and identify several lacunae in existing data collection approaches used to collect such dialogue. We present this survey, the first survey to focus exclusively on multi-party dialogue corpora, to motivate research in this area. Through our discussion of existing data collection methods, we identify desiderata and guiding principles for multi-party data collection to contribute further towards advancing this area of dialogue research.


Introduction
To say research in conversational agents and natural language generation has seen an explosive growth in recent years would be an understatement, as evidenced by the increasing number of papers published on this topic. However, most current research in this area has focused on two-party or dyadic conversations. This focus is important, since many open questions remain with dialogue systems in dyadic settings, such as modeling longterm dialogue context modeling and infusion of knowledge, persona and empathy (Li et al., 2016;Hedayatnia et al., 2020; Nevertheless, there is still a pressing need to focus on more naturally occurring conversations which consist of more than two speakers (Kirchhoff and Ostendorf, 2003), also known as multiparty dialogue. Humans naturally tend to work in groups and teams. Conversational agents capable of working in multi-party dialogue situations stand to advance the future of work, since they can be integrated into teams, e.g., in surgery, search and rescue, or manufacturing and design. The settings for such agents could be informal (e.g. chatroom assistants) or formal (e.g. meeting assistants) settings. Particularly, with conversational assistants such as Amazon Alexa, there is a push to develop AI to understand multiple users and act as teammates (Winkler et al., 2019;Seeber et al., 2020).
At the same time, methods and models built for two-party cannot simply be generalized for multi-party conversations. Some challenges that are unique to multi-party dialogue include speaker identification (figuring out who is speaking), turntaking (understanding whether to respond or not) and tailoring the content of the response to each agent or person (Sibun, 1997).
Several of these challenges can be approached through data-driven methods (Hawes et al., 2009;de Bayser et al., 2019). Given that corpora are the currency for data-driven methods, and facilitate further research on building data-driven multi-party dialogue systems, we present this systematic survey of existing corpora for multi-party dialogue. We describe how these corpora (Section 3) were collected (Section 4) along with the tasks that are undertaken on these corpora. Our key goal is to identify desiderata that could help guide data collection efforts towards making research in multi-party dialogue more mature (Section 5).
Our survey follows prior efforts in systematic reviews of dialogue corpora (Serban et al., 2018), evaluation of chatbots (Venkatesh et al., 2018;Deriu et al., 2020), and NLG evaluation (Howcroft et al., 2020). Gatt and Krahmer (2018) provide a meticulous survey of the state-of-the-art in Natural Language Generation, however they do not include a separate discussion on corpora. The systematic review of dialogue corpora conducted by Serban et al. (2018) does not primarily focus on multi-party corpora. Deriu et al. (2021) provide a systematic survey on the evaluation of dialogue systems, which includes a section of datasets and benchmarks, but again the focus is not primarily towards multi-party dialogue systems. Consequently, the goal of this article is to make the following contributions: • presenting a comprehensive listing of a large number of available multi-party dialogue corpora, and organize these into a taxonomy. To accomplish this goal, we start from a collection of over 300 published papers. • presenting a detailed overview of data collection methods for multi-party dialogue, especially the need for specialized equipment and environments. • providing recommendations for collecting new useful datasets, to advance research in this area. Our intent is that with an up-to-date synthesis of available resources, and by drawing attention to the challenges particular to multi-party dialogue, we can provide insights of exploiting recent datadriven techniques to address these challenges.

Method
Selection Criteria: Similar to recent work in systematic review of relevant literature, we followed the PRISMA method to identify, screen and include articles for this survey (Howcroft et al., 2020;Reiter, 2018). We searched Google Scholar and Semantic Scholar for the keywords multi-party dialogue and variations thereof (e.g., multi-party, multiparty conversation). We began by considering all papers that appeared in conferences and journals which focus on NLP and NLG, including all × CL venues as well as AI conferences and venues (e.g., AAAI, IJCNLP, Interspeech). We then iterated through the references and citations of these papers, and included any relevant articles that were missed through keyword search. This identification step resulted in 362 papers overall.
As part of our screening process, we limit the discussion to corpora that (a) have already been used in existing research in conversational systems; (b) which have a text component, and focus on the English language; and (c) which include multiple speakers in the majority of conversations, finally resulting in 343 papers. We release our annotated references to the 343 papers on Github 1 . Unsurprisingly, we found that majority of corpora papers were published in LREC and SIGDIAL venues, in addition to *ACL venues. 1 https://tiny.one/mpd-references Organizing corpora by genre: Next, we organized all included corpora into a new taxonomy ( Figure 1). Corpora are first categorized by whether they include Spoken or Written dialogue. Spoken corpora are further divided as unscripted vs. scripted. Within these type-based divisions, the corpora are then arranged by their main sources. The unscripted spoken corpora are thus arranged into 4 main categories -informal discourse mainly consisting of informal interactions such as radio talk shows, formal discourse mainly consisting of formal interactions such as debates, spontaneous speech mainly consisting of spontaneous interactions such as teenage talk, and meetings and interviews mainly focused on data from sources such as TV interviews. Similarly, the scripted spoken corpora are arranged into scripts and dialogues from plays, movies and TV series. Lastly, the written corpora are arranged into four categories-synchronous mainly consisting of chatroom talk, and online game-playing forums with users mainly conversing about game progression; and asynchronous mainly consisting of posts made on online forums and short text messages on microblog websites with character limits for posts.
Tables 1 and 2 present additional details about each corpus, including the name and source citation, topics presented, quantitative details such as number of dialogues, words, total length, and speakers, as well as whether they are multi-modal. All the available corpora have been used for datadriven research on multi-party dialogue. We thus include the Task Descriptions each corpus has been used for in the past. These tasks range from machine reading comprehension and turn-taking to speaker-identification.

Existing Corpora for Multi-Party Dialogue
In the subsections below, we outline the descriptions of each corpus.

Spoken Corpora
Spoken corpora is the most prevalent type of corpora available for multi-party dialogue. Spoken corpora presented in this paper are further divided into two main categories (Table 1) -unscripted which refers to spontaneous, unplanned dialogues; and scripted which refers to planned dialogue such as TV and movie scripts. The distinction between scripted and unscripted is made to allow for dif- ferent modelling tasks, since scripted dialogue displays an absence of hesitations, repetitions and other normal non-fluency features.

Unscripted Spoken Corpora
One of the earliest multi-party spoken corpora is the British National Corpus (BNC) (Leech, 1992), originally created by the Oxford University press in 1980s-1990s. Covering a wide range of genres, including some written conversations, as well as POS-tagged data (Leech et al., 1994), it is important as a generalized multi-party conversation corpus. It has been used to study social differentiation in the use of English vocabulary (Rayson et al., 1997), word frequency differences in spoken vs written text (Leech et al., 2001), and amplifiers such as "very" and "so" in the English language (Xiao and Tao, 2007). The Cambridge and Nottingham Corpus of Discourse in English (CANCODE) (McCarthy, 1998) focuses on interpersonal communication conversations in various settings such as hair salons and restaurants. It has been used to study language use for teaching in classrooms (O'keeffe et al., 2007), and is a resource for linguistic features of discourse. This corpus is not openly available anymore.
A more informal, casual English corpus is the Bergen Corpus of London Teenage Language (COLT) (Stenström and Breivik, 1993), which was recorded in secret to document spontaneous conversations and teenage language. It has been used to study trends in teenage language evolution (Stenström et al., 2002), and is an excellent resource for spontaneous informal multi-party interaction.
The D64 Multimodal corpus (Oertel et al., 2012) is another addition to spontaneous multi-party dialogue, focusing on recording multi-modal dynamic interactions without specifying a topic.
The COnversational Speech In Noisy Environments (COSINE) (Stupakov et al., 2012) corpus introduces data collected in noisy environments, extending the challenges faced in multi-party dialogue such as turn-taking, and has been used to evaluate such systems (Raffensperger et al., 2012).
The IDIAP Wolf corpus (Hung and Chittaranjan, 2010) focuses on group behavior in a competitive role-playing game setting, with a pre-condition of bad faith interactions similar to the "werewolf" or "mafia" game that makes it a unique corpus. It has been used in the AIWolfDial task to help train game-playing AI (Kano et al., 2019). While specific instances of lying are not annotated, the "werewolf" of each game is annotated in the corpus.
On the flip side, the TEAMS corpus (Litman et al., 2016) where teams of three or four speakers play two rounds of a cooperative board game, provides a novel resource for studying team entrainment and participation dominance. Rahimi and Litman (2020) use it to build a novel graph-based vector representation of multi-party entrainment, gaining insights into the dynamics of the entrainment relations.
Recently, the Critical Role Dungeons and Dragons Dataset (CRD3) (Rameshkumar and Bailey, 2020) was released, which is a game-based corpus set in an open-ended scenario. The paper also provides an abstractive summarization benchmark and evaluation, based on each dialogue's summary.
Within formal settings, one of the oldest corpus is the Corpus of Spoken, Professional American-English (CSPAE) (Barlow, 2000), consisting of two main components. The first is White House press conferences, and the second is transcripts of meetings on national tests involving statements, discussions, and questions. In the past, it has proved a valuable resource for studying idioms and their usage (Liu, 2003). It is available as a paid resource.
The Michigan Corpus of Academic Spoken English (MICASE) (Simpson-Vlach and Leicher, 2006) includes academic speech from university settings. It also comes with abstracts for each transcript, and has been used in online speech summarization (Murray and Renals, 2007).
Debate-based settings are also ideal candidates for multi-party corpora building, and thus the Intelligence Squared Debates (IQ2US) (Yang et al., 2010) are an important source. They follow an Oxford-style debating structure, and contain structured data making for a great resource for debate and argumentation analysis .
Canal9 (Vinciarelli et al., 2009) is another debate corpus, consisting of political debates. It includes a rich set of socially relevant annotations, and has been used in tasks such as conflict detection (Kim et al., 2012). A historic debate corpus is the Trial Proceedings component of the Corpus of English Dialogues (CED) (Kytö and Walker, 2006), which has been used to study signalling function in discourse (Lenker, 2018).
Supplementing formal discourse in debate corpora are formal meeting corpora, with 2 corpora that have become really important for studying multi-party decision-making and discussions of actions to take are the ICSI meeting corpus (Janin et al., 2003), which also has Meeting Recorder Dialogue Act (MRDA) annotations (Shriberg et al., 2004); and the multi-modal AMI meeting corpus . ICSI has been used to further study multi-party language modeling (Ji and Bilmes, 2004), and AMI has been used to build summarization for meetings (Zhu et al., 2020).
Recent additions include data from interviews, such as the INTERVIEW (Majumder et al., 2020) and MediaSum (Zhu et al., 2021) corpora. They include transcripts from interviews on channels such as National Public Radio NPR and CNN.

Scripted Spoken Corpora
Scripted spoken corpora consist of pre-defined scripts such as those for plays, movies, and TV series. These are inherently different as they are not spontaneous, and have pre-defined roles for speakers as well as information on when the dialogues turns are taken. Some corpora are actually labelled with this information, while others are simply transcript-like (Table 1).
One of the earliest available scripted spoken corpora is a second component of the Corpus of En-glish Dialogue CED (Kytö and Walker, 2006) focusing on Prose Fiction. It has been used to study language styles in Shakespeare's plays in the context of contemporaneous plays (Demmen, 2012).
The Movie-DiC Corpus (Banchs, 2012) consists of a wide range of American movie scripts, along with context descriptions. It has even been used to generate parallel corpora for dialogue translation (Wang et al., 2016). The Film Scripts Online Series corpus includes British movie scripts, but is not available online.
The Cornell Movie-Dialogue Corpus (Danescu-Niculescu-Mizil and Lee, 2011) contains metadata associated with each movie script, and has been used to generate emotionally aligned responses to dialogue (Asghar et al., 2020).
The Character Style From Film Corpus (Walker et al., 2012a) is another resource contributing towards guided text generation by providing character styles, created from the archive IMSDB. It has been used to generate stylistic dialogue for narratives (Xu et al., 2018).
Both the OpenSubtitles (Tiedemann, 2012) and SubTle corpus (Ameixa and Coheur, 2013) are based on the OpenSubtitles site. They are corpora of plain scripts, but the website continues to contribute as a resource for more data (Lison and Tiedemann, 2016;Lison et al., 2018).
Bridging the sources of movie and TV scripts is the Corpus of American Soap Operas (Davies, 2013) which focuses on informal language, and has been used to study cultural representation differences in American soap operas (Khaghaninejad et al., 2019).
A TV series corpus including data from shows like The Big Bang Theory and Game of Thrones, supplemented by crowd-sourced contributions for tasks such as summarization is the TVD Corpus (Roy et al., 2014). It has been used to build models for speaker identification (Knyazeva et al., 2015). The Serial Speakers (Bost et al., 2020) dataset supplements data from both the aforementioned TV serials by also including the House of Cards and additional annotations.
Recently, the Multimodal EmotionLines Dataset (MELD) (Poria et al., 2019) corpus has been presented by extending the (ELD) (Hsu et al., 2018), with audio-visual modality along with text. It has been used as a resource for Dialogue Act Classification (Saha et al., 2020). The MEISD (Firdaus et al., 2020) dataset is build further with TV scripts from 10 series, adding Friends, How I Met Your Mother, The Office, House M.D., Grey's Anatomy, Castle, Breaking Bad to the aforementioned series.

Written Corpora
Written corpora for multi-party have often resulted from online chatroom discussions, like the NPS Chat Corpus (Forsythand and Martell, 2007), which is shared as a part of the NLTK (Loper and Bird, 2002), and is one of the first Computer-Mediated corpora.
The Ubuntu IRC chatroom has also contributed to corpora such as the Ubuntu Dialogue Corpus (Lowe et al., 2015) and Ubuntu Chat Corpus (Uthus and Aha, 2013), which were collected as users asked questions relating to Ubuntu on the forum, and other users answered the questions. They have been used to train end-to-end dialogue systems (Lowe et al., 2017). The Molweni corpus  builds on the Ubuntu Chat Dialogue corpus, and adds annotations for machine reading comprehension and dscourse parsing.
Another corpus based on chatroom data is the Multi-Party Chat (MPC) Corpus (Shaikh et al., 2010) which presents an annotated corpus based on four levels with communication links, dialogue acts, local topics and meso-topics, and has been used to understand user roles and modeling leadership and influence .
Game-playing corpora such as the Settlers of Catan Corpus (Afantenos et al., 2012) and Cards Corpus (Djalali et al., 2011) are great informal additions to chatroom corpora, with a competitive environment albeit in an informal setting. They have been used for tasks such as training models for negotiation dialogues (Cadilhac et al., 2013).
Online forums such as Reddit, and Wikipedia have also contributed to such corpora. These notably include the Reddit (Chang et al., 2020) corpus which has also been extended into larger corpora (Baumgartner et al., 2020).
There have also been argumentative corpora obtained from online interactions, like the Reddit Domestic Abuse Corpus (Schrading et al., 2015) taken from subreddits specific on domestic abuse, allowing for discourse analysis on this subject.
Debate and agreement corpora such as the Internet Argument Corpus (Walker et al., 2012b), Agreement in Wikipedia Talk Pages (Andreas et al., 2012) and Agreement by Create Debaters (Rosenthal and McKeown, 2015), from debate and discussion forums online such as CreateDebate also contribute towards argumentation in dialogue research (Rakshit et al., 2018).
Additionally, there have been corpora obtained from social media such as UseNet and Twitter. These include the UseNet Corpus Westbury, 2007, 2011), a platform which is considered a precursor to more recent forums; and the Twitter Corpus (Ritter et al., 2010), which was intended to help model dialogue acts.

Special Mentions
This section includes special mentions of corpora as well as frameworks and toolkits that do not fall under our previous categories.
There are very few corpora which have focused on human-machine dialogue for multi-party interactions. The only such corpora existing to the best of our knowledge is the Mission Rehearsal Exercise (MRE) Corpus (Robinson et al., 2004), which presents a dataset built as audio face-toface sessions between human trainees and virtual agents. The main theme of the multimodal dataset is decision-making for a platoon-leader in a peacekeeping mission, with the trainee acting as a lieutenant. The corpora has about 30K words, 2K utterances, and a total of 55 speakers. Traum et al. (2008) also introduce another 3-party negotiation dialogue corpus, called the Stabilization and Support Operations (SASO-EN) corpus, which grew out of experiments on the MRE corpus (Lee et al., 2007), focusing on eye-gaze behavior in 3-party negotiation. In an example scenario, the data consists of a human user who plays the role of a captain whose mission is to move a local clinic to a safer location by negotiating with the doctor and mayor of the city.
FriendsPersona (Jiang et al., 2020) is a another scripted spoken multi-party corpus, which focuses on annotated personalities of scripted characters based on the Big Five personality traits, consisting of 711 conversations from the TV show Friends. It was recently introduced, and has already been used towards personality detection tasks (Christian et al., 2021;. In the formal meeting and lecture space, the IDIAP meeting corpus (Jovanovic et al., 2006) is another extension under the AMI project (AMI and ICSI were discussed in Section 3.1.1), which focuses on addressing behavior in multi-modal, multi-party, face-to-face conversations. The cor-pus additionally contains hand-annotated dialogue acts, adjacency pairs, addressees and gaze directions of meeting participants. The Computers in Human Interaction Loop (CHIL) is another corpus (Mostefa et al., 2007) which provides numerous synchronized audio and video streams of real lectures and meetings, captured in multiple recording sites over a period of 4 years, focusing on human interaction in smart rooms. However, this corpus is a paid resource, available via ELRA 2 . Connected to formal spoken corpora, but focusing on the question-answering task in multi-party dialogue is the recently introduced QAConv corpus (Wu et al., 2021), with 34k questions taken from about 28k dialogues, with around 26k words and 32 speakers consisting of conversations taken from email, panels and other formal communication channels.
There are also several corpora, especially multimodal, which have been transcribed, but we could not find the statistics. These include the VACE multimodal meeting corpus (Chen et al., 2005), which investigates the interaction among speech, gesture, posture, and gaze in meetings. Another corpus is the MULTISIMO corpus (Koutsombogera and Vogel, 2018), towards modeling of collaborative aspects of multimodal behavior in groups that perform simple tasks between 2 people, supported by a facilitator. Mana et al. (2007) also present the Mission Survival Corpora (MSC) 1 and 2, a multi-modal corpus of multi-party meetings, automatically annotated using audio-visual cues (speech rate, pitch and energy, head orientation, hand and body fidgeting). Due to the limited information available, we do not add these corpora to the tables or the taxonomy.
A variation of the Machines Talking to Machines framework (Shah et al., 2018) allows a simulated user bot and a domain-agnostic system bot to converse to exhaustively generate dialogue "outlines", i.e. sequences of template utterances and their semantic parses, which can then be contextually rewritten by crowdworkers to maintain saliency and coherence while preserving meaning. We include the framework in this survey as it could contribute to collecting data for multi-party dialogue by extending it to include more simulated users and bots.
We also make special mention of the Convokit tool (Chang et al., 2020), which is a toolkit for 2 https://tiny.one/chil-data downloading corpora for dialogues. It allows the downloads to follow standard format for all available corpora. It also provides the functionality to load custom datasets in a similar format, making it easier to work with multiple corpora at once.

Data Collection Methods
Several methods of data collection have been used to collect the aforementioned corpora. We organize these into three main categories and discuss in detail below.
Aggregated from various sources: BNC, CANCODE, and MICASE employ the aggregation method to build the corpora. They pull information from various sources, including text from sources such as newspapers, journals, publicly available government meetings, radio phone-ins, academic writings, seminars, advising sessions etc. These corpora incorporate multiple types of speech, and often include speech surrounding multiple topics (especially BNC and CANCODE, MICASE mainly focuses on academic settings to collect data). They are thus great candidates for studying language semantics and have been employed to study largescale vocabularies (McCarthy et al., 2010) and word sense disambiguation (Roberts and Erklärung, 2012) in the past.
Transcribed from pre-recorded media: Single (or double) source origins, such as COLT, CRD3, and IQ2, maintain focus on certain themes, such as formal meeting data. These are not collected within specialized environments, but consist of either transcribed speech recorded in the wild, transcribed interviews & meetings, and online forum or social media data. This category also includes scripted corpora, which are usually collections of various scripts & dialogues from plays, movies and TV series, such as TVD and SubTle. Having a set theme allows these corpora to be used for generating themed text such as MELD being used for character identification as a part of the 2018 SemEval challenge (Choi and Chen, 2018).
Collected in specialized environments: Most multi-modal corpora employ specialized environments or equipment to collect data that can be synchronized across multiple modalities. Most focus on data collection using audio, which can then be transcribed. Specialized room environments with studio-quality recording (ICSI, AMI), closetalking mics (ICSI, IDIAP Wolf, TEAMS), and a combination of far-and close-field mics (COSINE,   AMI) have provided better data collection for corpora, allowing for annotations of speech activity and pauses as well. Another popular data collection method focuses on video, such as motion sensing (D64), and video cams (IDIAP Wolf, TEAMS, AMI), which supplement speech data well by also allowing for annotation of head movement, gesture, and eye-gaze tracking.
There are also multiple projects that emulate online social media platforms for controlled data collection, such as the Truman platform and Community Connect (Mahajan et al., 2021).

Desiderata for Data Collection
Given the multitude of corpora available and the modeling tasks that need to be undertaken to develop conversational agents for multi-party dialogue, we outline here three key criteria for future efforts in data collection: 1. Participant balance and tracking: We find from the tasks identified in Tables 1 and 2 that speaker and addressee identification are important open tasks in multi-party dialogue modeling. Con-sequently, corpora should contain sufficient information, in the data or in the metadata, to track participants within dialogues and across dialogues, if possible. Where possible, participants should be balanced in terms of age, gender and ethnicity and other demographic factors, so as to not preferentially model any specific type of language use.
2. Signal to Noise ratio: The corpora should contain a sufficiently high number of texts as possible, however, these should be of sufficiently high quality. Particularly, for data that are scraped from the web (e.g. Twitter or Reddit), it is possible for the noise to drown out important signals in the data. It is important to document all considerations and assumptions made when collecting the data. In most cases, specific details are outlined for data that are collected under specialized settings, and extreme care is taken to synchronize collection across modalities. We encourage a similar level of attention to detail when data are aggregated from existing sources. When possible, data collection studies should be preregistered so that researchers can describe their hypotheses, methods, and analy-ses beforehand (Nosek et al., 2019).
3. Ethical Considerations: Creating corpora focusing on multiple speakers requires multiple considerations to protect personally identifiable information (PII), while making sure that the corpus is annotated well to allow for usability. Especially in the case of multi-modal corpora, where eye-gaze and head movements have been used as features for tasks such as turn-taking, there are important guidelines to consider since it is not possible to remove PII easily (Benedict et al., 2019).

Discussion
The three desiderata listed above provide us with a set of guidelines for thinking about the challenges for thoughtful data collection. This (potentially non-exhaustive) list of questions is inspired by the current movement in several research fields to preregister studies in advance (Nosek et al., 2019;Vilhuber, 2020) and the needs for datasheets for datasets (Gebru et al., 2018). Research

Conclusion and Future Work
We present a systematic review and a taxonomy of available corpora for multi-party dialogue. We also identify key tasks that are typically conducted through the use of these corpora and we review how existing corpora are collected. To ensure that datadriven models that are developed using these and any future corpora, are high quality, we advance three critical desiderata, that lead us to several guiding principles. While we attempt to be as comprehensive as possible, there are certain limitations of this present article. We recognize that our review focuses entirely on English language data and models. Certainly, corpora exist in other languages, e.g. in Chinese and French (Riou et al., 2015;. We also do not provide any detail about the modeling tasks, e.g. turn taking. Extending our review to include additional languages and detailed description of modeling tasks is indeed part of a future, larger publication.