license:mit size_categories:n<1K format:csv modality:tabular modality:text library:datasets library:pandas library:mlcroissant library:polars region:us
---
license: mit
---

## Dataset Overview

This dataset was collected as part of the project "[**IMBUE: Improving Interpersonal Effectiveness through Simulation and Just-in-time Feedback with Human-Language Model Interaction**](https://aclanthology.org/2024.acl-long.47/)." 

We are releasing this dataset with the hope that it provides valuable opportunities for researchers to develop and evaluate new LLM-based tools for interpersonal skill training across a range of fields, including natural language processing, conversational AI, and computational psychology.

For any questions or concerns, please reach out to ilin@cs.washington.edu 

## Data Collection Methods 
### Interpersonal Effectiveness Framework 
The dataset leverages the DEAR MAN interpersonal effectiveness framework, a widely used framework from Dialectical Behavioral Therapy (DBT) that teaches conversational strategies and emotional regulation.

DEAR MAN stands for: **D**escribe, **E**xpress, **A**ssert, **R**einforce, Staying **M**indful, **A**ppear Confident, and **N**egotiate.

### Conversations on bespoke difficult situations between mTurk participants and simulated conversation partner 

- Each participant contributed three conversations, one from each category: family, social, and work.

- Participants engaged in conversations with a large language model (LLM) trained to simulate a conversation partner within the participant's self-defined bespoke, difficult situations.

- Participants were asked to have conversations that were at least 10 responses long or until the simulated partner agreed with them.

- Before writing each message, participants chose one or more DEAR MAN strategies to guide their communication.

Screenshots of the conversation collection interface are provided in the paper (Appendix J).

### Expert annotations on conversational and emotional skill use
The dataset includes annotations from six clinical experts trained in Dialectical Behavior Therapy (DBT) who actively use DEAR MAN in their practice. 

Expert Selection Criteria: The researchers specifically recruited experts who are clinical psychology PhD, PostDoc, or practitioners, and only selected those who indicated they "sometimes" or "regularly" work with clients on DEAR MAN skills. This selection process ensured the annotators had practical experience applying the DEAR MAN framework in therapeutic settings.

Annotation Process: Experts annotated the conversations using an interactive tool. Each message was assessed for the following:
- Skill Identification: Experts identified which DEAR MAN skills were present in each utterance.
- Skill Use Rating: Each skill used in an utterance was rated as either "strong" or "weak". Mindfulness and confidence were rated as "yes" or "no."
- Improvement Suggestions: Experts provided specific suggestions for improving "weak" utterances and offered rewritten versions demonstrating "strong" skill use. 
- Additionally, experts suggested skills for utterances where they believed a skill should have been applied but wasn't and provided rewritten utterances demonstrating their suggestions.

Screenshots of the annotation interface are provided in the paper (Appendix L).

## Privacy and Ethical Considerations

The researchers obtained approval from the Institutional Review Board (IRB) at the University of Washington before conducting the study. All researchers involved in the study completed human subjects protection training and received IRB certification.

Informed Consent: Informed consent was obtained from all participants involved in both the data collection and user study phases. Participants were informed that:
- They would be interacting with an AI-based model simulating a conversation partner.
- The data they provided would be used for research purposes.
- The model's responses might contain upsetting content due to the nature of the simulated difficult conversations.

Data Privacy: The study did not collect any Personally Identifiable Information (PII). Participants were asked to avoid including PII in the situations they described or the conversations they had with the model. The collected conversations and situation descriptions were manually filtered to remove any identifiable names or locations.

Crisis Resources: Although content filters were used to minimize potentially harmful outputs from the model, participants were provided with links to Crisis Text Line and the 988 Suicide and Crisis Lifeline as a precaution.


For further details on the methodology and findings, refer to the associated paper.



## Dataset Schema


The following table provides a brief description of each column in the dataset:

| Column Name                | Description                                                                                      |
|---------------------------|--------------------------------------------------------------------------------------------------|
| `conversation_id`         | A unique ID assigned to this conversation/situation.                                            |
| `situation`               | A brief description of the participant-provided situation, written by the participant.           |
| `category`                | Participant-chosen category of their situation, among family, social, or work.                  |
| `message_id`              | The message ID in each conversation, ordered sequentially.                                       |
| `message_text`            | Message written by the participants.                                                             |
| `label_describe`          | Expert's annotation of whether and how well the participant used the "Describe" skill.             |
| `suggestion_describe`     | Expert's suggestion on how to improve the use of the "Describe" skill.                          |
| `rewrite_describe`        | Expert's rewritten message based on their own suggestion for the "Describe" skill.              |
| `label_express`           | Expert's annotation of whether and how well the participant used the "Express" skill.              |
| `suggestion_express`      | Expert's suggestion on how to improve the use of the "Express" skill.                           |
| `rewrite_express`         | Expert's rewritten message based on their own suggestion for the "Express" skill.               |
| `label_assert`            | Expert's annotation of whether and how well the participant used the "Assert" skill.               |
| `suggestion_assert`       | Expert's suggestion on how to improve the use of the "Assert" skill.                            |
| `rewrite_assert`          | Expert's rewritten message based on their own suggestion for the "Assert" skill.                |
| `label_reinforce`         | Expert's annotation of whether and how well the participant used the "Reinforce" skill.           |
| `suggestion_reinforce`    | Expert's suggestion on how to improve the use of the "Reinforce" skill.                         |
| `rewrite_reinforce`       | Expert's rewritten message based on their own suggestion for the "Reinforce" skill.             |
| `label_negotiate`         | Expert annotation of whether and how well the participant used the "Negotiate" skill.           |
| `suggestion_negotiate`    | Expert's suggestion on how to improve the use of the "Negotiate" skill.                         |
| `rewrite_negotiate`       | Expert's rewritten message based on their own suggestion for the "Negotiate" skill.             |
| `username_expert`         | A randomly-assigned username of the expert who provided the annotation and suggestions.                         |
| `label_confident`         | Expert's annotation of whether and how well the participant demonstrated the "Appear Confident" skill in their communication. |
| `suggestion_confident`    | Expert's suggestion on how to enhance the "Appear Confident" skill in communication.                               |
| `rewrite_confident`       | Expert's rewritten message based on their own suggestion to enhance the "Appear Confident" skille.                  |
| `label_mindful`           | Expert's annotation of whether and how well the participant demonstrated the "Stay Mindful" skill in their communication. |
| `suggestion_mindful`      | Expert's suggestion on how to enhance the "Stay Mindful" skill in communication.                              |
| `rewrite_mindful`         | Expert's rewritten message based on their own suggestion to enhance the "Stay Mindful" skill.                 |
