SCROLLS: Standardized CompaRison Over Long Language Sequences
  NLP benchmarks have largely focused on short texts, such as sentences and
paragraphs, even though long texts comprise a considerable amount of natural
language in the wild. We introduce SCROLLS, a suite of tasks that require
reasoning over long texts. We examine existing long-text datasets, and handpick
ones where the text is naturally long, while prioritizing tasks that involve
synthesizing information across the input. SCROLLS contains summarization,
question answering, and natural language inference tasks, covering multiple
domains, including literature, science, business, and entertainment. Initial
baselines, including Longformer Encoder-Decoder, indicate that there is ample
room for improvement on SCROLLS. We make all datasets available in a unified
text-to-text format and host a live leaderboard to facilitate research on model
architecture and pretraining methods.
