size_categories:n<1K format:csv modality:tabular modality:text library:datasets library:pandas library:mlcroissant library:polars arxiv:2005.00614 region:us
---
TODO: "Add YAML tags here. Delete these instructions and copy-paste the YAML tags obtained with the online tagging app: https://huggingface.co/spaces/huggingface/datasets-tagging"
YAML tags: "Find the full spec here: https://github.com/huggingface/hub-docs/blob/main/datasetcard.md?plain=1"
---

# Dataset Card Creation Guide

## Table of Contents
- [Dataset Card Creation Guide](#dataset-card-creation-guide)
  - [Table of Contents](#table-of-contents)
  - [Dataset Description](#dataset-description)
    - [Dataset Summary](#dataset-summary)
    - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
    - [Languages](#languages)
  - [Dataset Structure](#dataset-structure)
    - [Data Instances](#data-instances)
    - [Data Fields](#data-fields)
    - [Data Splits](#data-splits)
  - [Dataset Creation](#dataset-creation)
    - [Curation Rationale](#curation-rationale)
    - [Source Data](#source-data)
      - [Initial Data Collection and Normalization](#initial-data-collection-and-normalization)
      - [Who are the source language producers?](#who-are-the-source-language-producers)
    - [Annotations](#annotations)
      - [Annotation process](#annotation-process)
      - [Who are the annotators?](#who-are-the-annotators)
    - [Personal and Sensitive Information](#personal-and-sensitive-information)
  - [Considerations for Using the Data](#considerations-for-using-the-data)
    - [Social Impact of Dataset](#social-impact-of-dataset)
    - [Discussion of Biases](#discussion-of-biases)
    - [Other Known Limitations](#other-known-limitations)
  - [Additional Information](#additional-information)
    - [Dataset Curators](#dataset-curators)
    - [Licensing Information](#licensing-information)
    - [Citation Information](#citation-information)
    - [Contributions](#contributions)

<!--## Dataset Description

- **Homepage:** [Add homepage URL here if available (unless it's a GitHub repository)]()
- **Repository:** [If the dataset is hosted on github or has a github homepage, add URL here]()
- **Paper:** [If the dataset was introduced by a paper or there was a paper written describing the dataset, add URL here (landing page for Arxiv paper preferred)]()
- **Leaderboard:** [If the dataset supports an active leaderboard, add link here]()
- **Point of Contact:** [If known, name and email of at least one person the reader can contact for questions about the dataset.]()
-->

### Dataset Summary
The Israeli-Palestinian-Conflict dataset is an English-language dataset contains manually collected claims, regarding the Israel-Palestine conflict, annotated both objectively with multi-labels to categorize the content according to common themes in such arguments, and subjectively by their level of impact on a moderately informed citizen. The primary purpose of this dataset is to support Israeli public relations efforts at all levels. 

<!--Briefly summarize the dataset, its intended use and the supported tasks. Give an overview of how and why the dataset was created. The summary should explicitly mention the languages present in the dataset (possibly in broad terms, e.g. *translations between several pairs of European languages*), and describe the domain, topic, or genre covered.-->


<!--### Supported Tasks and Leaderboards

For each of the tasks tagged for this dataset, give a brief description of the tag, metrics, and suggested models (with a link to their HuggingFace implementation if available). Give a similar description of tasks that were not covered by the structured tag set (repace the `task-category-tag` with an appropriate `other:other-task-name`).

- `task-category-tag`: The dataset can be used to train a model for [TASK NAME], which consists in [TASK DESCRIPTION]. Success on this task is typically measured by achieving a *high/low* [metric name](https://huggingface.co/metrics/metric_name). The ([model name](https://huggingface.co/model_name) or [model class](https://huggingface.co/transformers/model_doc/model_class.html)) model currently achieves the following score. *[IF A LEADERBOARD IS AVAILABLE]:* This task has an active leaderboard which can be found at [leaderboard url]() and ranks models based on [metric name](https://huggingface.co/metrics/metric_name) while also reporting [other metric name](https://huggingface.co/metrics/other_metric_name).
-->

### Languages

The text in the dataset is in English, as spoken by Reddit users on the [r/IsraelPalestine](https://www.reddit.com/r/IsraelPalestine/s/wcCjJ94VtF) and [r/ChangeMyView](https://www.reddit.com/r/changemyview/s/RxCmmNwDdh) subreddits, and by youtube transcribed from English debates. The associated BCP-47 code is en.

<!--Provide a brief overview of the languages represented in the dataset. Describe relevant details about specifics of the language such as whether it is social media text, African American English,...

When relevant, please provide [BCP-47 codes](https://tools.ietf.org/html/bcp47), which consist of a [primary language subtag](https://tools.ietf.org/html/bcp47#section-2.2.1), with a [script subtag](https://tools.ietf.org/html/bcp47#section-2.2.3) and/or [region subtag](https://tools.ietf.org/html/bcp47#section-2.2.4) if available.-->


## Dataset Structure

### Data Instances

<!--Provide an JSON-formatted example and brief description of a typical instance in the dataset. If available, provide a link to further examples.-->

A typical data point comprises a unique id (example_id), the type of phase in the annotation process (batch), the origin split from which the data point came (split)
a claim about the Israeli-Palestinian conflict (text), the annotators annotation vectors for the 1'st task (annotator[i]_t1_label) and the majority vote vector label (t1_label), the annotators annotation on a scale of 1 to 5 for the 2'st task (annotator[i]_t2_label) and the evaluated final label (t2_label),

An example from the Israeli-Palestinian-Conflict dev set looks as follows:

```
{
  "example_id": 212,
  "batch": "exploration",
  "split": "validation",
  "text": "You're very biased and hypocrites!! Hamas is the byproduct of 80years of occupation and genocide, Israel is responsible for this atrocities don't come here blaming Palestinians and why they support Hamas,bc Hamas is all they have to fight with occupation force, calling them terrorists will not change the truth even Mandela was once called a terrorist by the colonial imperialists! Gtfo",
  "annotator1_t1_label": [0, 1, 1, 0, 1, 0, 0, 1, 0, 0],
  "annotator2_t1_label": [0, 1, 1, 0, 1, 0, 0, 1, 1, 0],
  "annotator3_t1_label": [0, 1, 1, 0, 1, 0, 0, 1, 0, 0],
  "annotator1_t2_label": 2,
  "annotator2_t2_label": 2,
  "annotator3_t2_label": 3,
  "t1_label": [0, 1, 1, 0, 1, 0, 0, 1, 0, 0],
  "t2_label": 2
}

```

<!--Provide any additional information that is not covered in the other sections about the data here. In particular describe any relationships between data points and if these relationships are made explicit.-->

### Data Fields

<!--List and describe the fields present in the dataset. Mention their data type, and whether they are used as input or output in any of the tasks the dataset currently supports. If the data has span indices, describe their attributes, such as whether they are at the character level or word level, whether they are contiguous or not, etc. If the datasets contains example IDs, state whether they have an inherent meaning, such as a mapping to other datasets or pointing to relationships between data points.

- `example_field`: description of `example_field`

Note that the descriptions can be initialized with the **Show Markdown Data Fields** output of the [Datasets Tagging app](https://huggingface.co/spaces/huggingface/datasets-tagging), you will then only need to refine the generated descriptions.-->

-  "example_id": an int claim's unique identifier without inherent meaning
-  "batch": a string representing the type of phase in the annotation process (detailed later in this card)
-  "split": a string representing the split this claim originally from 
-  "text": a string which is the collected claim regarding the Israel-Palestine conflict
-  "t1_label": a list of size 10, representing the majority vote of the annotators annotation binary vector for the 1'st task, where every entrance represent a feature that is present/not present in the claim, respectivly as follows:
    - "Advocacy for Israel" - Statements that express support or positive sentiment towards Israel. These statements advocate for Israel’s actions, policies, or general stance in the conflict
    - "Advocacy for Palestine" - Statements that express support or positive sentiment towards Palestine. These statements advocate for Palestine’s actions, policies, or general stance in the conflict. Notice, this excludes Hamas or any terror organizations affiliated with the palestinians.
    - "Criticism of Israel" - Statements that express opposition or negative sentiment towards Israel. These statements criticize Israel’s actions, policies, government, IDF or any other offical institution
    - "Criticism of Palestine" - Statements that express opposition or negative sentiment towards Palestine. These statements criticize Palestine’s actions, policies, or general stance in the conflict.
    - "Pro-terror organization" - Statements that express support for organizations considered to be terrorist groups. These statements advocate for or justify the actions and policies of these groups.
    - "Anti-terror organization" - Statements that express opposition or negative sentiment towards terror organization . These statements criticize the terror organization actions, policies, or general stance in the conflict..
    - "Backed-Up" - Statements that provide evidence, data, or references to support their claims. These statements may cite sources, statistics, or authoritative opinions to validate their arguments.
    - "Logical Argument" - Statements that present a logical and reasoned case or position on the Israel-Palestine conflict. These statements typically include premises and a conclusion, structured to persuade or convince others of a particular viewpoint. They often address counterarguments and provide rationale for their stance.
    - "Moral argument" - A moral argument is a statement that appeals to ethical principles, values, and beliefs about right and wrong to support a particular stance or viewpoint. These arguments often invoke notions of justice, human rights, fairness, and compassion. They aim to persuade by highlighting the moral implications and ethical considerations of the issue at hand, rather than relying solely on factual evidence or logical reasoning.
    - "Figurative language" - Statements that use figurative language to describe the conflict or parties involved. These statements use metaphors and comparisons and ”perspective changes” to illustrate points vividly.

-  "t2_label": the rate a claim is impactful on a scale from 1 to 5, defined as the extent to which the statement influences, engages, or resonates with a moderately informed citizen (A person with a basic to moderate understanding of current events and historical context, familiar with the general facts of the Israel-Palestine conflict, but without a strong personal or ideological bias). This rating considers factors such as clarity, emotional appeal, relevance, and informativeness, with an emphasis on maintaining objectivity, regardless of personal beliefs or biases. The final annotation for this subjective task is based on [Aroyo and Welty, 2015](https://www.researchgate.net/publication/283092501_Truth_Is_a_Lie_Crowd_Truth_and_the_Seven_Myths_of_Human_Annotation)
-  "annotator1_t1_label": a list of size 10, representing the 1st annotator annotation binary vector for the 1'st task
-  "annotator2_t1_label": a list of size 10, representing the 2nd annotator annotation binary vector for the 1'st task
-  "annotator3_t1_label": a list of size 10, representing the 3rd annotator annotation binary vector for the 1'st task
-  "annotator1_t2_label": an int representing the 1st annotator label for the 2'nd task
-  "annotator2_t2_label": an int representing the 2nd annotator label for the 2'nd task
-  "annotator3_t2_label": an int representing the 3rd annotator label for the 2'nd task


### Data Splits
<!--Describe and name the splits in the dataset if there are more than one.Describe any criteria for splitting the data, if used. If there are differences between the splits (e.g. if the training annotations are machine-generated and the dev and test ones are created by humans, or if different numbers of annotators contributed to each example), describe them here.Provide the sizes of each split. As appropriate, provide any descriptive statistics for the features, such as average length.  For example:-->
|                         | train | validation/dev | test |
|-------------------------|------:|-----------:|-----:|
| # Sentences             |  210  |     37     |  150 |
| Average Sentence Length | 90.45 |    97.38   | 85.55|

## Dataset Creation

### Curation Rationale

<!--What need motivated the creation of this dataset? What are some of the reasons underlying the major choices involved in putting it together?-->

The primary purpose of this dataset is to support Israeli public relations efforts at all levels. Specifically, it can aid the Israeli Ministry of Public Diplomacy and organizations dedicated to promoting Israeli advocacy. In light of the events of the past year (2023), we gained insight into the critical importance of Israel’s ability to effectively present public arguments and monitor public opinion. This dataset is valuable for developing models to evaluate the quality of arguments prior to addressing international bodies like the UN, the ICJ, and the US Congress. Additionally, it can be utilized for large scale investigating of social media, helping to strategically focus efforts in shaping public opinion.

### Source Data
<!--This section describes the source data (e.g. news text and headlines, social media posts, translated sentences,...)-->
YouTube-Transcribed Debates: We downloaded transcripts from debates and used ChatGPT to extract participants' short arguments, manually selecting them for inclusion.
Reddit: Comments about the Israel-Palestine conflict were manually extracted from the subreddits [r/IsraelPalestine](https://www.reddit.com/r/IsraelPalestine/s/wcCjJ94VtF) and [r/ChangeMyView](https://www.reddit.com/r/changemyview/s/RxCmmNwDdh).

#### Initial Data Collection and Normalization

<!--Describe the data collection process. Describe any criteria for data selection or filtering. List any key words or search terms used. If possible, include runtime information for the collection process.

If data was collected from other pre-existing datasets, link to source here and to their [Hugging Face version](https://huggingface.co/datasets/dataset_name).

If the data was modified or normalized after being collected (e.g. if the data is word-tokenized), describe the process and the tools used.-->

Our approach during the manual selection of arguments was to include diverse characteristics, such as varying lengths, languages, and structures, while ensuring a balanced representation of both sides of the conflict. None of the arguments in our dataset have been modified.

#### Who are the source language producers?

The language producers are users of the  [r/IsraelPalestine](https://www.reddit.com/r/IsraelPalestine/s/wcCjJ94VtF) and [r/ChangeMyView](https://www.reddit.com/r/changemyview/s/RxCmmNwDdh) subreddits. No further demographic information was available from the data source.
<!--State whether the data was produced by humans or machine generated. Describe the people or systems who originally created the data.

If available, include self-reported demographic or identity information for the source data creators, but avoid inferring this information. Instead state that this information is unknown. See [Larson 2017](https://www.aclweb.org/anthology/W17-1601.pdf) for using identity categories as a variables, particularly gender.

Describe the conditions under which the data was created (for example, if the producers were crowdworkers, state what platform was used, or if the data was found, what website the data was found on). If compensation was provided, include that information here.

Describe other people represented or mentioned in the data. Where possible, link to references for the information. -->

### Annotations

<!--If the dataset contains annotations which are not part of the initial data collection, describe them in the following paragraphs.-->

#### Annotation process

<!--If applicable, describe the annotation process and any tools used, or state otherwise. Describe the amount of data annotated, if not all. Describe or reference annotation guidelines provided to the annotators. If available, provide interannotator statistics. Describe any annotation validation processes.-->

Our annotation process was conducted in several distinct phases, aimed at creating a comprehensive dataset. Below is a detailed overview of our workflow:

#### **Initial Setup and Guidelines**
We began by collecting a total of approximately 400 documents, which were divided into three distinct batches:
- **Exploration Batch**: 50 documents used for the creation and iterative refinement of the annotation guidelines. Each group member annotated this set independently, and we identified areas of disagreement and refined our categories accordingly. We discussed our disagreements and reached a consensus on the approach for each scenario. These decisions were then codified in our guidelines.
- **Evaluation Batch**: 80 documents used to evaluate the inter-annotator agreement (IAA) after creating the first draft of the guidelines. During this phase, group members independently annotated the documents without discussing their annotations to prevent collaboration that could influence the IAA scores.
- **Part 3 Batch**: ~270 documents reserved for later annotation by a third-party group. This phase focused on creating guidelines that could be followed by external annotators, ensuring consistency and reliability across annotations.

Once the guidelines were finalized, two annotators revisited and independently annotated the entire Exploration Batch. Although this batch had previously been used for guideline development, it was fully annotated to ensure a complete dataset for training and evaluation.

#### **Peer Feedback and Guideline Improvement**
In this phase, we received feedback on our guidelines from a third-party team:
- The third-party team annotated a sample of 50 documents from the **Part 3 - Exploration Batch** using our guidelines. They provided feedback on areas where the guidelines could be improved.
- We incorporated their feedback to address ambiguities and ensure the clarity of our instructions.
- Finally, the third-party team annotated a larger sample of ~270 documents (**Part 3 - Evaluation Batch**) using the refined guidelines. This helped further validate the robustness of the guidelines.

#### Inter-Annotator Agreement (IAA)

To evaluate the reliability of the annotations, we calculated IAA scores across the different phases and groups involved.
The following table summarizes the scores:

| Group    | Phase  | #   | Label1 | Label2 | Label3 | Label4 | Label5 | Label6 | Label7 | Label8 | Label9 | Label10 | MAE   | Corr  |
|----------|--------|-----|--------|--------|--------|--------|--------|--------|--------|--------|--------|--------|-------|-------|
| Creators | Explor | 50  | 0.69   | 0.67   | 0.71   | 0.66   | NAN    | 0.04   | NAN    | 0.28   | 0.21   | 0.50   |       |       |
| Creators | Eval   | 80  | 0.64   | 0.53   | 0.76   | 0.66   | NAN    | 0.72   | 0.29   | 0.30   | 0.40   | 0.47   | 1.024 | 0.264 |
| Peer     | Explor | 100 | 0.84   | 0.51   | 0.78   | 0.60   | NAN    | 0.70   | 0.66   | 0.58   | 0.57   | 0.31   |       |       |
| Peer     | Eval   | 167 | 0.62   | 0.64   | 0.70   | 0.67   | NAN    | 0.66   | 0.72   | 0.65   | 0.71   | 0.74   | 0.740 | 0.698 |

- For the first task the Fleiss' Kappa metric was used to evaluate the multi-label agreement between annotators for each label (Label1 through Label10) in the objective task. The higher Kappa values indicate a stronger agreement, with the peer group achieving a higher overall Kappa score in the Exploration Batch.

- For the subjective task, we calculated the Mean Absolute Error (MAE) and Pearson Correlation (Corr) between the annotators’ judgments, with the peer group achieving a better overall MAE and Correlation.


#### Who are the annotators?

<!--If annotations were collected for the source data (such as class labels or syntactic parses), state whether the annotations were produced by humans or machine generated.

Describe the people or systems who originally created the annotations and their selection criteria if applicable.

If available, include self-reported demographic or identity information for the annotators, but avoid inferring this information. Instead state that this information is unknown. See [Larson 2017](https://www.aclweb.org/anthology/W17-1601.pdf) for using identity categories as a variables, particularly gender.

Describe the conditions under which the data was annotated (for example, if the annotators were crowdworkers, state what platform was used, or if the data was found, what website the data was found on). If compensation was provided, include that information here.-->

The dataset was annotated by two groups, all humans. The creators group, consisting of 2 women and 2 men. The peer group, consisting of 2 women and a man. All annotators were aged 22-28 and were Israeli-Jewish. There was no compensation provided.

### Personal and Sensitive Information

<!--State whether the dataset uses identity categories and, if so, how the information is used. Describe where this information comes from (i.e. self-reporting, collecting from profiles, inferring, etc.). See [Larson 2017](https://www.aclweb.org/anthology/W17-1601.pdf) for using identity categories as a variables, particularly gender. State whether the data is linked to individuals and whether those individuals can be identified in the dataset, either directly or indirectly (i.e., in combination with other data).

State whether the dataset contains other data that might be considered sensitive (e.g., data that reveals racial or ethnic origins, sexual orientations, religious beliefs, political opinions or union memberships, or locations; financial or health data; biometric or genetic data; forms of government identification, such as social security numbers; criminal history).  

If efforts were made to anonymize the data, describe the anonymization process. -->

This data contains religious beliefs and political opinions but is completely anonymize.

## Considerations for Using the Data

### Social Impact of Dataset

<!--Please discuss some of the ways you believe the use of this dataset will impact society.

The statement should include both positive outlooks, such as outlining how technologies developed through its use may improve people's lives, and discuss the accompanying risks. These risks may range from making important decisions more opaque to people who are affected by the technology, to reinforcing existing harmful biases (whose specifics should be discussed in the next section), among other considerations.

Also describe in this section if the proposed dataset contains a low-resource or under-represented language. If this is the case or if this task has any impact on underserved communities, please elaborate here.-->

The use of this dataset has the potential to foster greater understanding of the Israeli-Palestinian conflict 
by providing data that can be used for research and technologies like sentiment analysis or claim validation tools, 
which may inform public discourse. These technologies can also aid in reducing misinformation and 
helping individuals engage more critically with content related to this imporatant conflict. 
However, there is a risk that automated systems built using this dataset might reinforce biases present in the data, potentially leading to one-sided analyses. 
Additionally, the complexity of the conflict means that decisions informed by these technologies might lack the nuance required for sensitive issues, 
impacting world outcomes in ways that are not easily understood by the affected populations, like classifying ones claim as pro-terror and the possible outcomes.

### Discussion of Biases

<!--Provide descriptions of specific biases that are likely to be reflected in the data, and state whether any steps were taken to reduce their impact.

For Wikipedia text, see for example [Dinan et al 2020 on biases in Wikipedia (esp. Table 1)](https://arxiv.org/abs/2005.00614), or [Blodgett et al 2020](https://www.aclweb.org/anthology/2020.acl-main.485/) for a more general discussion of the topic.

If analyses have been run quantifying these biases, please add brief summaries and links to the studies here.-->

The significant bias we created during the annotation process stems from approaching the Israeli-Palestinian conflict from an Israeli-Jewish perspective. 
We attempted to minimize this bias by designing the first task to be as objective as possible. 
This included creating balanced categories and naming them neutrally, while carefully explaining the meaning of each category in the guidelines in an impartial way. 
For the second task, we framed the instructions to ask annotators to assess the claims from the viewpoint of a "moderately informed citizen," 
defined as someone familiar with the general facts of the conflict but without strong personal or ideological bias. 
Despite our efforts, we acknowledge that the dataset inevitably contains some bias, 
but we believe its creation is crucial given the importance and relevance of the topic.

### Other Known Limitations
[N/A]
<!--If studies of the datasets have outlined other limitations of the dataset, such as annotation artifacts, please outline and cite them here.-->

## Additional Information

### Dataset Curators
Affiliation: Data Science students @ Technion, we got no funding.
<!--List the people involved in collecting the dataset and their affiliation(s). If funding information is known, include it here.-->

### Licensing Information
[N/A]
Provide the license and link to the license webpage if available.

### Citation Information

<!--Provide the [BibTex](http://www.bibtex.org/)-formatted reference for the dataset. For example:-->
```
@article{article_id,
  author    = {Avishag Nevo, Bella Perel, Hadar Sugarman, Tomer Shigani},
  title     = {The Israeli-Palestinian Conflict Dataset},
  year      = {2024}
}
```

<!--If the dataset has a [DOI](https://www.doi.org/), please provide it here.-->

### Contributions

Thanks to Prof. Roi Reichart and Mr. Nitay Calderon for guiding us in this dataset process.

