Drawing out of Distribution with Neuro-Symbolic Generative Models
Learning general-purpose representations from perceptual inputs is a hallmark
of human intelligence. For example, people can write out numbers or characters,
or even draw doodles, by characterizing these tasks as different instantiations
of the same generic underlying process -- compositional arrangements of
different forms of pen strokes. Crucially, learning to do one task, say
writing, implies reasonable competence at another, say drawing, on account of
this shared process. We present Drawing out of Distribution (DooD), a
neuro-symbolic generative model of stroke-based drawing that can learn such
general-purpose representations. In contrast to prior work, DooD operates
directly on images, requires no supervision or expensive test-time inference,
and performs unsupervised amortised inference with a symbolic stroke model that
better enables both interpretability and generalization. We evaluate DooD on
its ability to generalise across both data and tasks. We first perform
zero-shot transfer from one dataset (e.g. MNIST) to another (e.g. Quickdraw),
across five different datasets, and show that DooD clearly outperforms
different baselines. An analysis of the learnt representations further
highlights the benefits of adopting a symbolic stroke model. We then adopt a
subset of the Omniglot challenge tasks, and evaluate its ability to generate
new exemplars (both unconditionally and conditionally), and perform one-shot
classification, showing that DooD matches the state of the art. Taken together,
we demonstrate that DooD does indeed capture general-purpose representations
across both data and task, and takes a further step towards building general
and robust concept-learning systems.