Target Transfer Q-Learning and Its Convergence Analysis
  Q-learning is one of the most popular methods in Reinforcement Learning (RL).
Transfer Learning aims to utilize the learned knowledge from source tasks to
help new tasks to improve the sample complexity of the new tasks. Considering
that data collection in RL is both more time and cost consuming and Q-learning
converges slowly comparing to supervised learning, different kinds of transfer
RL algorithms are designed. However, most of them are heuristic with no
theoretical guarantee of the convergence rate. Therefore, it is important for
us to clearly understand when and how will transfer learning help RL method and
provide the theoretical guarantee for the improvement of the sample complexity.
In this paper, we propose to transfer the Q-function learned in the source task
to the target of the Q-learning in the new task when certain safe conditions
are satisfied. We call this new transfer Q-learning method target transfer
Q-Learning. The safe conditions are necessary to avoid the harm to the new
tasks and thus ensure the convergence of the algorithm. We study the
convergence rate of the target transfer Q-learning. We prove that if the two
tasks are similar with respect to the MDPs, the optimal Q-functions in the
source and new RL tasks are similar which means the error of the transferred
target Q-function in new MDP is small. Also, the convergence rate analysis
shows that the target transfer Q-Learning will converge faster than Q-learning
if the error of the transferred target Q-function is smaller than the current
Q-function in the new task. Based on our theoretical results, we design the
safe condition as the Bellman error of the transferred target Q-function is
less than the current Q-function. Our experiments are consistent with our
theoretical founding and verified the effectiveness of our proposed target
transfer Q-learning method.
