language:en size_categories:n<1K format:csv modality:tabular modality:text library:datasets library:pandas library:mlcroissant library:polars region:us
---
language:
- en
---

# Dataset Card for Reddit Bullying Advice Dataset

This dataset consists of Reddit posts related to bullying, along with advice provided by both Reddit users and large language models, annotated for tasks including content categorization, advice clarity evaluation, and the identification of human versus LLM-generated responses.

## Dataset Details

### Dataset Description

This dataset contains posts and comments from Reddit, specifically focused on bullying. The posts and responses include advice, some of which were provided by Reddit users and others generated by various LLMs. The dataset includes three primary tasks: (1) an objective multilabel categorization task of the content, (2) a subjective task assessing the practicality of the advice, and (3) a task to determine whether a response was written by a human or an LLM.


- **Curated by:** Or Cohen, Keren Gruteke-Klein, Or Dado, Tal Shalom
- **Language(s) (NLP):** English
- **License:** Unknown

### Dataset Sources
- **Repository:** https://huggingface.co/datasets/OrKositsky/bullying-advices

## Uses


### Direct Use

This dataset can be used for building and evaluating models in three key tasks:

1. Objective Task (multilabel): Determines which categories the comment falls into out of the next categories: involve others, self actions, showing support, change enviorment, involve bully or victim.
2. Subjective Task: Each comment is individually rated based on its practicality and specificity using the following scale:
   
   5 - Highly Practical and Specific Advice
   
   4 - Practical but Vague Advice
   
   3 - Somewhat Practical Advice
   
   2 - Impractical or Unrealistic Advice
   
   1 - No Advice Given
   
4. Perceived Human vs LLM Identification: A subjective classification task where annotators judge whether a comment appears to have been written by a human or generated by a large language model (LLM). This task is based on human perception rather than the actual source of the comment.


### Out-of-Scope Use

This dataset is not suitable for applications that could involve the exploitation of sensitive information or activities that cause harm or violate ethical guidelines.
## Dataset Structure

The dataset contains the following key columns:

### Key Columns:
- example_id: Unique identifier for each example.
- batch: Indicates the batch of data collection - exploration, evaluation or part 3 (annotated by different team - more on that later).
- split: Specifies whether the example is in the train, dev, or test set.
- original_text: Original Reddit post and comment.
- text: The preprocessed text that was eventually tagged.
- t1_majority_label: The majority label for Task 1 (Objective Categorization).
- t2_majority_label: The majority label for Task 2 (Subjective Practicality and Clarity Evaluation).
- t3_majority_label: The majority label for Task 3 (Perceived Human vs LLM Identification).
- annotator1_t1_label, annotator2_t1_label, ..., annotator4_t1_label: Labels from individual annotators for Task 1.
- annotator1_t2_label, annotator2_t2_label, ..., annotator4_t2_label: Labels from individual annotators for Task 2.
- annotator1_t3_label, annotator2_t3_label, ..., annotator4_t3_label: Labels from individual annotators for Task 3.
- comment_source: Indicates whether the response was written by a human or an LLM.
- post_length: Length of the original post.
- comment_length: Length of the comment.

### Dataset Split
The dataset is structured in several splits:

- Train
- Dev
- Test
  
Each split includes annotations for three tasks:
- Objective Categorization Task (t1_majority_label): A multilabel task to categorize the content of posts and responses.
- Subjective Practicality Task (t2_majority_label): A subjective task that evaluates the specifity and practicality of the advice given, with labels reflecting the majority opinion of annotators.
- Perceived Human/LLM Identification Task (t3_majority_label): A subjective task where annotators indicate whether they believe the comment was written by a human or generated by an LLM, based on their perception rather than the actual source.

## Dataset Creation

### Curation Rationale
The dataset was created to analyze how online communities, particularly Reddit, handle sensitive topics like bullying and how well advice, both human and AI-generated, performs in this context. The aim is to create models that can better understand the nuances of online advice and provide helpful insights.

### Source Data


#### Data Collection and Processing
The posts and comments were collected from Reddit. Additional data generated by LLMs was used to complement user-generated responses. Manual annotation was applied to categorize the data, evaluate advice quality, and classify the authorship.


#### Who are the source data producers?

The dataset contains data from Reddit users and content generated by large language models. Annotations were manually added by human annotators to provide labels for the three tasks.

### Annotations

#### Annotation process

The dataset was annotated by two groups of last-year data science students. The annotation process followed these steps:

**Exploratory Phase:** 
The students from the first group initially annotated 100 "exploration" documents to create consensus on labeling. All group members annotated the same 25 documents, discussed discrepancies, and iterated on the annotation guidelines.

**Evaluation Phase:** 
A batch of 100 documents was annotated independently by all 4 annotators in the first group, without any collaboration, using the finalized guidelines. 

**Final Annotation Phase:** 
The remaining 300 documents, labeled as "part 3," were annotated by a separate group of 3 students. The guidelines created by the first group were tested in this phase, with annotations made according to the finalized instructions.

**Measurements:**

**Objective Task:**

Evaluation Phase:
- Agreement rate	89.3%
- Cohen’s Kappa Score	0.72
- Fleiss Kappa Score	0.73

Part 3:
- Agreement rate	85.1%
- Cohen’s Kappa Score	0.56
- Fleiss Kappa Score	0.55

**Subjective Task:**

Evaluation Phase:
- Cohen’s Kappa Score 0.11
- Spearman Correlation 0.77
- Pearson Correlation 0.75
- MAE 0.54
- Fleiss Kappa Score 0.36

Part 3:
- Cohen’s Kappa Score 0.42
- Spearman Correlation 0.86
- Pearson Correlation 0.87
- MAE 0.58
- Fleiss Kappa Score 0.41

**LLM Task:**

Evaluation Phase:
- Agreement rate	0.55
- Cohen’s Kappa Score	0.10
- Fleiss Kappa Score	0.34

Part 3:
- Agreement rate	0.61
- Cohen’s Kappa Score	0.14
- Fleiss Kappa Score	0.13

**Annotation Guidelines Link:**
https://docs.google.com/document/d/1OSwy09dwg_6BhQNVfO9hdlHmjAO8a9tC/edit?usp=sharing&ouid=104573220305918809590&rtpof=true&sd=true.

These guidelines, iterated through the exploration phase, were detailed to ensure that annotators could label new documents independently. For clarity, examples were included for each type of label. 
Majority voting was used for determining the final label when discrepancies occurred.

#### Who are the annotators?

The annotators are final-year data science students with no expertise in psychology or professional advice, divided into two groups:
- The first group created the annotation guidelines using the exploration batch and then annotated the evaluation phase samples.
- The second group annotated the 'part 3' batch samples, following the provided annotation guidelines.

#### Personal and Sensitive Information
The dataset contains discussions around sensitive topics, specifically bullying, which may be considered personal or private. Efforts were made to anonymize the data by removing personal identifiers to protect the privacy of individuals, though it is still possible that the nature of the discussions may reveal sensitive content.

## Bias, Risks, and Limitations
This dataset may contain biases as it reflects the personal opinions and generated content of both Reddit users and LLMs, which may vary in accuracy, quality, and helpfulness. Additionally, the advice provided by anonymous users and models may not always align with professional or expert guidance. 

Furthermore, the evaluation samples were annotated by the same individuals who manually collected the data and the LLM responses. This overlap in data collection and annotation could introduce bias, as the annotators might have been influenced by their familiarity with the content or preconceived notions during the annotation process.
### Recommendations

Users should be cautious when using the dataset to generalize findings, as biases from both user-generated content and model-generated responses could affect outcomes. Additionally, manual annotations, performed by annotators with limited subject matter expertise, may introduce bias into the dataset. Therefore, the dataset is best utilized in a controlled environment where moderation and bias mitigation techniques are applied.

## Dataset Card Authors
Or Cohen, Keren Gruteke-Klein, Or Dado, Tal Shalom


## Dataset Card Contact

For more informaction, please contact cohen.or@campus.technion.ac.il