Learning to Synthesize Programs as Interpretable and Generalizable
  Policies
  Recently, deep reinforcement learning (DRL) methods have achieved impressive
performance on tasks in a variety of domains. However, neural network policies
produced with DRL methods are not human-interpretable and often have difficulty
generalizing to novel scenarios. To address these issues, prior works explore
learning programmatic policies that are more interpretable and structured for
generalization. Yet, these works either employ limited policy representations
(e.g. decision trees, state machines, or predefined program templates) or
require stronger supervision (e.g. input/output state pairs or expert
demonstrations). We present a framework that instead learns to synthesize a
program, which details the procedure to solve a task in a flexible and
expressive manner, solely from reward signals. To alleviate the difficulty of
learning to compose programs to induce the desired agent behavior from scratch,
we propose to first learn a program embedding space that continuously
parameterizes diverse behaviors in an unsupervised manner and then search over
the learned program embedding space to yield a program that maximizes the
return for a given task. Experimental results demonstrate that the proposed
framework not only learns to reliably synthesize task-solving programs but also
outperforms DRL and program synthesis baselines while producing interpretable
and more generalizable policies. We also justify the necessity of the proposed
two-stage learning scheme as well as analyze various methods for learning the
program embedding.
