Adaptive and Multiple Time-scale Eligibility Traces for Online Deep
  Reinforcement Learning
  Deep reinforcement learning (DRL) is one promising approach to teaching
robots to perform complex tasks. Because methods that directly reuse the stored
experience data cannot follow the change of the environment in robotic problems
with a time-varying environment, online DRL is required. The eligibility traces
method is well known as an online learning technique for improving sample
efficiency in traditional reinforcement learning with linear regressors rather
than DRL. The dependency between parameters of deep neural networks would
destroy the eligibility traces, which is why they are not integrated with DRL.
Although replacing the gradient with the most influential one rather than
accumulating the gradients as the eligibility traces can alleviate this
problem, the replacing operation reduces the number of reuses of previous
experiences. To address these issues, this study proposes a new eligibility
traces method that can be used even in DRL while maintaining high sample
efficiency. When the accumulated gradients differ from those computed using the
latest parameters, the proposed method takes into account the divergence
between the past and latest parameters to adaptively decay the eligibility
traces. Bregman divergences between outputs computed by the past and latest
parameters are exploited due to the infeasible computational cost of the
divergence between the past and latest parameters. In addition, a generalized
method with multiple time-scale traces is designed for the first time. This
design allows for the replacement of the most influential adaptively
accumulated (decayed) eligibility traces.
