task_categories:text-generation task_categories:text-classification language:en license:apache-2.0 size_categories:10K<n<100K format:csv modality:tabular modality:text library:datasets library:dask library:mlcroissant library:polars arxiv:2211.17148 region:us
---
license: apache-2.0
task_categories:
- text-generation
- text-classification
language:
- en
size_categories:
- 1K<n<10K
configs:
- config_name: bona_fide
  data_files: "cleaned_hotel_goal_dia.csv"
- config_name: missing_sentence_completion
  data_files: "Missing_Sentence_*.csv"
- config_name: next_response_generation
  data_files: "Next_Response_*.csv"
- config_name: goal_to_dialogue
  data_files: "G2D_*.csv"
- config_name: paraphrase_system
  data_files: "Par_chatbot_system_*.csv"
- config_name: paraphrase_system_user
  data_files: "Par_full_chatbot_*.csv"
- config_name: end_to_end_conversation
  data_files: "E2E_Convo_*.csv"
---

# SPADE: Structured Prompting Augmentation for Dialogue Enhancement in Machine-Generated Text Detection

<!-- Provide a quick summary of the dataset. -->

SPADE contains a repository of customer service line synthetic user dialogues with goals, augmented from MultiWOZ 2.1 using GPT-3.5 and Llama 70B. 
The datasets are intended for training and evaluating machine generated text detectors in dialogue settings.

There are 15 English datasets generated using 5 different augmentation methods and 2 large language models, including:
- **Bona Fide** (cleaned MultiWOZ 2.1 labelled by ConvLab-3): *cleaned_hotel_goal_dia.csv*
- **Missing Sentence Completion**: *Missing_Sentence_gpt.csv*, *Missing_Sentence_llama.csv*
- **Next Response Generation**: *Next_Response_gpt.csv*, *Next_Response_llama.csv*
- **Goal to Dialogue**: *G2D_gpt.csv*, *G2D_llama.csv*
- **Paraphrase Dialogue (synthetic system)**: *Par_chatbot_system_gpt.csv*, *Par_chatbot_system_llama.csv*
- **Paraphrase Dialogue (synthetic system and user)**: *Par_full_chatbot_gpt.csv*, *Par_full_chatbot_llama.csv*
- **End-to-End Conversation**:
    - gpt3.5 system gpt3.5 user: *E2E_Convo_gpt_gpt.csv*
    - gpt3.5 system llama user: *E2E_Convo_gpt_llama.csv*
    - llama system llama user: *E2E_Convo_llama_llama.csv*
    - llama system gpt3.5 user: *E2E_Convo_llama_gpt.csv*



## Dataset Structure

<!-- This section provides a description of the dataset fields, and additional information about the dataset structure such as criteria used to create the splits, relationships between data points, etc. -->

The datasets are of csv file format and contain the following columns:
| Dataset                    | Column            | Description                                                                                              |
|----------------------------|-------------------|----------------------------------------------------------------------------------------------------------|
| **All**                     | *dia_no* / *new_dia_no* | Unique ID for each dialogue. Dialogues with the same *dia_no* across synthetic datasets are based on the bona fide dialogue with the same *new_dia_no*. |
|                            | *dia*             | The dialogue itself, either bona fide or synthetic.                                                      |
| **Bona Fide**               | *new_goal*        | The cleaned user goal associated with the dialogue.                                                      |
| **Next Response Generation**| *turn_no*         | Zero-based turn number of the user response within the dialogue.                                         |
|                            | *context*         | Context provided to the user for generating the next response.                                           |
|                            | *response*        | Single utterance response generated by the user based on the provided context.                           |



## Dataset Creation

### Source Data
The source data is the hotel-domained subset of the [MultiWOZ 2.1](https://aclanthology.org/2020.lrec-1.53/) dataset. 
The MultiWOZ 2.1 dataset contains customer service dialogues like hotel booking, 
collected using a Wizard-of-Oz setup where two participants act as the user and system. 
This dataset with [ConvLab-3](https://arxiv.org/abs/2211.17148) labelled goals is used as our baseline for applying data augmentation frameworks.


#### Data Preprocessing

<!-- This section describes the data collection and processing process such as data selection criteria, filtering and normalization methods, tools and libraries used, etc. -->
The source data is preprocessed to resolve goal-dialogue mismatches, including missing or incorrect goals. This includes a two-step refinement:
1. Llama 70B automatically verified goal achievement.
2. manually revised goals to ensure alignment without changing dialogue content. Incomplete dialogues were removed, resulting in a final set of 616 out of 623 refined hotel dialogues (Bona Fide dataset).

#### Data Collection
The data augmentation frameworks and data collection process applied to generate the synthetic datasets are detailed in the paper (to be updated).