TOMA: Topological Map Abstraction for Reinforcement Learning
  Animals are able to discover the topological map (graph) of surrounding
environment, which will be used for navigation. Inspired by this biological
phenomenon, researchers have recently proposed to generate graph representation
for Markov decision process (MDP) and use such graphs for planning in
reinforcement learning (RL). However, existing graph generation methods suffer
from many drawbacks. One drawback is that existing methods do not learn an
abstraction for graphs, which results in high memory and computation cost. This
drawback also makes generated graph non-robust, which degrades the planning
performance. Another drawback is that existing methods cannot be used for
facilitating exploration which is important in RL. In this paper, we propose a
new method, called topological map abstraction (TOMA), for graph generation.
TOMA can generate an abstract graph representation for MDP, which costs much
less memory and computation cost than existing methods. Furthermore, TOMA can
be used for facilitating exploration. In particular, we propose planning to
explore, in which TOMA is used to accelerate exploration by guiding the agent
towards unexplored states. A novel experience replay module called vertex
memory is also proposed to improve exploration performance. Experimental
results show that TOMA can outperform existing methods to achieve the
state-of-the-art performance.
