R\'enyi State Entropy for Exploration Acceleration in Reinforcement
  Learning
  One of the most critical challenges in deep reinforcement learning is to
maintain the long-term exploration capability of the agent. To tackle this
problem, it has been recently proposed to provide intrinsic rewards for the
agent to encourage exploration. However, most existing intrinsic reward-based
methods proposed in the literature fail to provide sustainable exploration
incentives, a problem known as vanishing rewards. In addition, these
conventional methods incur complex models and additional memory in their
learning procedures, resulting in high computational complexity and low
robustness. In this work, a novel intrinsic reward module based on the R\'enyi
entropy is proposed to provide high-quality intrinsic rewards. It is shown that
the proposed method actually generalizes the existing state entropy
maximization methods. In particular, a $k$-nearest neighbor estimator is
introduced for entropy estimation while a $k$-value search method is designed
to guarantee the estimation accuracy. Extensive simulation results demonstrate
that the proposed R\'enyi entropy-based method can achieve higher performance
as compared to existing schemes.
