The Web Is Your Oyster - Knowledge-Intensive NLP against a Very Large
  Web Corpus
  In order to address increasing demands of real-world applications, the
research for knowledge-intensive NLP (KI-NLP) should advance by capturing the
challenges of a truly open-domain environment: web-scale knowledge, lack of
structure, inconsistent quality and noise. To this end, we propose a new setup
for evaluating existing knowledge intensive tasks in which we generalize the
background corpus to a universal web snapshot. We investigate a slate of NLP
tasks which rely on knowledge - either factual or common sense, and ask systems
to use a subset of CCNet - the Sphere corpus - as a knowledge source. In
contrast to Wikipedia, otherwise a common background corpus in KI-NLP, Sphere
is orders of magnitude larger and better reflects the full diversity of
knowledge on the web. Despite potential gaps in coverage, challenges of scale,
lack of structure and lower quality, we find that retrieval from Sphere enables
a state of the art system to match and even outperform Wikipedia-based models
on several tasks. We also observe that while a dense index can outperform a
sparse BM25 baseline on Wikipedia, on Sphere this is not yet possible. To
facilitate further research and minimise the community's reliance on
proprietary, black-box search engines, we share our indices, evaluation metrics
and infrastructure.
