Addressing "Documentation Debt" in Machine Learning Research: A
  Retrospective Datasheet for BookCorpus
  Recent literature has underscored the importance of dataset documentation
work for machine learning, and part of this work involves addressing
"documentation debt" for datasets that have been used widely but documented
sparsely. This paper aims to help address documentation debt for BookCorpus, a
popular text dataset for training large language models. Notably, researchers
have used BookCorpus to train OpenAI's GPT-N models and Google's BERT models,
even though little to no documentation exists about the dataset's motivation,
composition, collection process, etc. We offer a preliminary datasheet that
provides key context and information about BookCorpus, highlighting several
notable deficiencies. In particular, we find evidence that (1) BookCorpus
likely violates copyright restrictions for many books, (2) BookCorpus contains
thousands of duplicated books, and (3) BookCorpus exhibits significant skews in
genre representation. We also find hints of other potential deficiencies that
call for future research, including problematic content, potential skews in
religious representation, and lopsided author contributions. While more work
remains, this initial effort to provide a datasheet for BookCorpus adds to
growing literature that urges more careful and systematic documentation for
machine learning datasets.
