license:mit arxiv:2310.13189 region:us
---
license: mit
viewer: false
---
# ScreenEval
contact: Barrett Lattimer, blattimer@asapp.com \
paper: [Fast and Accurate Factual Inconsistency Detection Over Long Documents](https://arxiv.org/abs/2310.13189) \
github: [scale-score](https://github.com/asappresearch/scale-score)

ScreenEval is a a novel dataset designed for factual inconsistency detection in long dialogues. 
52 TV transcripts were summarized by humans, Longformer, and GPT-4, then each summary sentence was labelled for factual consistency with the source TV transcript.
Additionally, if a summary sentence was factually consistent, labellers provided relevant utterance support in the source document.
ScreenEval is the longest dialogue based dataset by tokens for factual inconsistency detection available to date.
The dialogue domain presents unique challenges such as long-distance coreference resolution and significant noise between relevant utterances.
ScreenEval is the dataset proposed in the paper "Fast and Accurate Factual Inconsistency Detection Over Long Documents" from EMNLP2023. 

## Stats at a glance
- 52 TV transcripts
- \>6k tokens per TV transcript
- 624 summary sentences in total (from humans, Longformer, and GPT-4)
- Relevant utterance labels for all factually consistent summary sentences

## Arguments
The following keys can be used to access the different part of the ScreenEval dataset.
| Key  | Type | Description |
| ------ | ------ | ------ |
| original_convo | List[str] | The source document that is to be summarized as a string |
| convo | List[List[str]] | The source document that is to be summarized split into a list of utterances |
| inferred_summary | List[str] | The summary sentence that is paired with the given source document |
| summary_id | List[str] | The source model for the summary sentence |
| convo_id | List[int] | The ID of the source document |
| annotated_summary | List[str] | The entire associated summary, with the focus summary sentence surrounded by `<mark><\mark>`|
| prediction_annotated_source_doc | List[str] | Raw source document |
| agreement | List[float] | Annotator agreement on summary sentence facutal inconsistency label |
| agg_label | List[bool] | Factual inconsistency label (true -> factually consistent, false -> factually inconsistent) |
| rel_utt | List[List[int]] | The indices of related utterances in the corresponding `convo` list.