Pre-Trained Language Models for Interactive Decision-Making
  Language model (LM) pre-training is useful in many language processing tasks.
But can pre-trained LMs be further leveraged for more general machine learning
problems? We propose an approach for using LMs to scaffold learning and
generalization in general sequential decision-making problems. In this
approach, goals and observations are represented as a sequence of embeddings,
and a policy network initialized with a pre-trained LM predicts the next
action. We demonstrate that this framework enables effective combinatorial
generalization across different environments and supervisory modalities. We
begin by assuming access to a set of expert demonstrations, and show that
initializing policies with LMs and fine-tuning them via behavior cloning
improves task completion rates by 43.6% in the VirtualHome environment. We then
examine how our framework may be used in environments without pre-collected
expert data. To do this, we integrate an active data gathering procedure into
pre-trained LMs. The agent iteratively learns by interacting with the
environment, relabeling the language goal of past 'failed' experiences, and
updating the policy in a self-supervised loop. The active data gathering
procedure also enables effective combinatorial generalization, outperforming
the best baseline by 25.1%. Finally, we explain these results by investigating
three possible factors underlying the effectiveness of the LM-based policy. We
find that sequential input representations (vs. fixed-dimensional feature
vectors) and favorable weight initialization are both important for
generalization. Surprisingly, however, the format of the policy inputs encoding
(e.g. as a natural language string vs. an arbitrary sequential encoding) has
little influence. Together, these results suggest that language modeling
induces representations that are useful for modeling not just language, but
also goals and plans.
