Generative Models are Unsupervised Predictors of Page Quality: A
  Colossal-Scale Study
  Large generative language models such as GPT-2 are well-known for their
ability to generate text as well as their utility in supervised downstream
tasks via fine-tuning. Our work is twofold: firstly we demonstrate via human
evaluation that classifiers trained to discriminate between human and
machine-generated text emerge as unsupervised predictors of "page quality",
able to detect low quality content without any training. This enables fast
bootstrapping of quality indicators in a low-resource setting. Secondly,
curious to understand the prevalence and nature of low quality pages in the
wild, we conduct extensive qualitative and quantitative analysis over 500
million web articles, making this the largest-scale study ever conducted on the
topic.
