Bongard-LOGO: A New Benchmark for Human-Level Concept Learning and
  Reasoning
  Humans have an inherent ability to learn novel concepts from only a few
samples and generalize these concepts to different situations. Even though
today's machine learning models excel with a plethora of training data on
standard recognition tasks, a considerable gap exists between machine-level
pattern recognition and human-level concept learning. To narrow this gap, the
Bongard problems (BPs) were introduced as an inspirational challenge for visual
cognition in intelligent systems. Despite new advances in representation
learning and learning to learn, BPs remain a daunting challenge for modern AI.
Inspired by the original one hundred BPs, we propose a new benchmark
Bongard-LOGO for human-level concept learning and reasoning. We develop a
program-guided generation technique to produce a large set of
human-interpretable visual cognition problems in action-oriented LOGO language.
Our benchmark captures three core properties of human cognition: 1)
context-dependent perception, in which the same object may have disparate
interpretations given different contexts; 2) analogy-making perception, in
which some meaningful concepts are traded off for other meaningful concepts;
and 3) perception with a few samples but infinite vocabulary. In experiments,
we show that the state-of-the-art deep learning methods perform substantially
worse than human subjects, implying that they fail to capture core human
cognition properties. Finally, we discuss research directions towards a general
architecture for visual reasoning to tackle this benchmark.
