A Taxonomy of Similarity Metrics for Markov Decision Processes
  Although the notion of task similarity is potentially interesting in a wide
range of areas such as curriculum learning or automated planning, it has mostly
been tied to transfer learning. Transfer is based on the idea of reusing the
knowledge acquired in the learning of a set of source tasks to a new learning
process in a target task, assuming that the target and source tasks are close
enough. In recent years, transfer learning has succeeded in making
Reinforcement Learning (RL) algorithms more efficient (e.g., by reducing the
number of samples needed to achieve the (near-)optimal performance). Transfer
in RL is based on the core concept of similarity: whenever the tasks are
similar, the transferred knowledge can be reused to solve the target task and
significantly improve the learning performance. Therefore, the selection of
good metrics to measure these similarities is a critical aspect when building
transfer RL algorithms, especially when this knowledge is transferred from
simulation to the real world. In the literature, there are many metrics to
measure the similarity between MDPs, hence, many definitions of similarity or
its complement distance have been considered. In this paper, we propose a
categorization of these metrics and analyze the definitions of similarity
proposed so far, taking into account such categorization. We also follow this
taxonomy to survey the existing literature, as well as suggesting future
directions for the construction of new metrics.
