Learning Task-General Representations with Generative Neuro-Symbolic
  Modeling
  People can learn rich, general-purpose conceptual representations from only
raw perceptual inputs. Current machine learning approaches fall well short of
these human standards, although different modeling traditions often have
complementary strengths. Symbolic models can capture the compositional and
causal knowledge that enables flexible generalization, but they struggle to
learn from raw inputs, relying on strong abstractions and simplifying
assumptions. Neural network models can learn directly from raw data, but they
struggle to capture compositional and causal structure and typically must
retrain to tackle new tasks. We bring together these two traditions to learn
generative models of concepts that capture rich compositional and causal
structure, while learning from raw data. We develop a generative neuro-symbolic
(GNS) model of handwritten character concepts that uses the control flow of a
probabilistic program, coupled with symbolic stroke primitives and a symbolic
image renderer, to represent the causal and compositional processes by which
characters are formed. The distributions of parts (strokes), and correlations
between parts, are modeled with neural network subroutines, allowing the model
to learn directly from raw data and express nonparametric statistical
relationships. We apply our model to the Omniglot challenge of human-level
concept learning, using a background set of alphabets to learn an expressive
prior distribution over character drawings. In a subsequent evaluation, our GNS
model uses probabilistic inference to learn rich conceptual representations
from a single training image that generalize to 4 unique tasks, succeeding
where previous work has fallen short.
