Learning and Evaluating General Linguistic Intelligence
  We define general linguistic intelligence as the ability to reuse previously
acquired knowledge about a language's lexicon, syntax, semantics, and pragmatic
conventions to adapt to new tasks quickly. Using this definition, we analyze
state-of-the-art natural language understanding models and conduct an extensive
empirical investigation to evaluate them against these criteria through a
series of experiments that assess the task-independence of the knowledge being
acquired by the learning process. In addition to task performance, we propose a
new evaluation metric based on an online encoding of the test data that
quantifies how quickly an existing agent (model) learns a new task. Our results
show that while the field has made impressive progress in terms of model
architectures that generalize to many tasks, these models still require a lot
of in-domain training examples (e.g., for fine tuning, training task-specific
modules), and are prone to catastrophic forgetting. Moreover, we find that far
from solving general tasks (e.g., document question answering), our models are
overfitting to the quirks of particular datasets (e.g., SQuAD). We discuss
missing components and conjecture on how to make progress toward general
linguistic intelligence.
