Component Transfer Learning for Deep RL Based on Abstract
  Representations
  In this work we investigate a specific transfer learning approach for deep
reinforcement learning in the context where the internal dynamics between two
tasks are the same but the visual representations differ. We learn a
low-dimensional encoding of the environment, meant to capture summarizing
abstractions, from which the internal dynamics and value functions are learned.
Transfer is then obtained by freezing the learned internal dynamics and value
functions, thus reusing the shared low-dimensional embedding space. When
retraining the encoder for transfer, we make several observations: (i) in some
cases, there are local minima that have small losses but a mismatching
embedding space, resulting in poor task performance and (ii) in the absence of
local minima, the output of the encoder converges in our experiments to the
same embedding space, which leads to a fast and efficient transfer as compared
to learning from scratch. The local minima are caused by the reduced degree of
freedom of the optimization process caused by the frozen models. We also find
that the transfer performance is heavily reliant on the base model; some base
models often result in a successful transfer, whereas other base models often
result in a failing transfer.
