Trajectory-wise Multiple Choice Learning for Dynamics Generalization in
  Reinforcement Learning
  Model-based reinforcement learning (RL) has shown great potential in various
control tasks in terms of both sample-efficiency and final performance.
However, learning a generalizable dynamics model robust to changes in dynamics
remains a challenge since the target transition dynamics follow a multi-modal
distribution. In this paper, we present a new model-based RL algorithm, coined
trajectory-wise multiple choice learning, that learns a multi-headed dynamics
model for dynamics generalization. The main idea is updating the most accurate
prediction head to specialize each head in certain environments with similar
dynamics, i.e., clustering environments. Moreover, we incorporate context
learning, which encodes dynamics-specific information from past experiences
into the context latent vector, enabling the model to perform online adaptation
to unseen environments. Finally, to utilize the specialized prediction heads
more effectively, we propose an adaptive planning method, which selects the
most accurate prediction head over a recent experience. Our method exhibits
superior zero-shot generalization performance across a variety of control
tasks, compared to state-of-the-art RL methods. Source code and videos are
available at https://sites.google.com/view/trajectory-mcl.
