CURI: A Benchmark for Productive Concept Learning Under Uncertainty
  Humans can learn and reason under substantial uncertainty in a space of
infinitely many concepts, including structured relational concepts ("a scene
with objects that have the same color") and ad-hoc categories defined through
goals ("objects that could fall on one's head"). In contrast, standard
classification benchmarks: 1) consider only a fixed set of category labels, 2)
do not evaluate compositional concept learning and 3) do not explicitly capture
a notion of reasoning under uncertainty. We introduce a new few-shot,
meta-learning benchmark, Compositional Reasoning Under Uncertainty (CURI) to
bridge this gap. CURI evaluates different aspects of productive and systematic
generalization, including abstract understandings of disentangling, productive
generalization, learning boolean operations, variable binding, etc.
Importantly, it also defines a model-independent "compositionality gap" to
evaluate the difficulty of generalizing out-of-distribution along each of these
axes. Extensive evaluations across a range of modeling choices spanning
different modalities (image, schemas, and sounds), splits, privileged auxiliary
concept information, and choices of negatives reveal substantial scope for
modeling advances on the proposed task. All code and datasets will be available
online.
