Controlling Conditional Language Models with Distributional Policy
  Gradients
  Machine learning is shifting towards general-purpose pretrained generative
models, trained in a self-supervised manner on large amounts of data, which can
then be applied to solve a large number of tasks. However, due to their generic
training methodology, these models often fail to meet some of the downstream
requirements (e.g. hallucination in abstractive summarization or wrong format
in automatic code generation). This raises an important question on how to
adapt pre-trained generative models to a new task without destroying its
capabilities. Recent work has suggested to solve this problem by representing
task-specific requirements through energy-based models (EBMs) and approximating
these EBMs using distributional policy gradients (DPG). Unfortunately, this
approach is limited to unconditional distributions, represented by
unconditional EBMs. In this paper, we extend this approach to conditional tasks
by proposing Conditional DPG (CDPG). We evaluate CDPG on three different
control objectives across two tasks: summarization with T5 and code generation
with GPT-Neo. Our results show that fine-tuning using CDPG robustly moves these
pretrained models closer towards meeting control objectives and -- in contrast
with baseline approaches -- does not result in catastrophic forgetting.
