CLUES: A Benchmark for Learning Classifiers using Natural Language
  Explanations
  Supervised learning has traditionally focused on inductive learning by
observing labeled examples of a task. In contrast, humans have the ability to
learn new concepts from language. Here, we explore training zero-shot
classifiers for structured data purely from language. For this, we introduce
CLUES, a benchmark for Classifier Learning Using natural language ExplanationS,
consisting of a range of classification tasks over structured data along with
natural language supervision in the form of explanations. CLUES consists of 36
real-world and 144 synthetic classification tasks. It contains crowdsourced
explanations describing real-world tasks from multiple teachers and
programmatically generated explanations for the synthetic tasks. To model the
influence of explanations in classifying an example, we develop ExEnt, an
entailment-based model that learns classifiers using explanations. ExEnt
generalizes up to 18% better (relative) on novel tasks than a baseline that
does not use explanations. We delineate key challenges for automated learning
from explanations, addressing which can lead to progress on CLUES in the
future. Code and datasets are available at: https://clues-benchmark.github.io.
