Realistic Actor-Critic: A Framework for Balance Between Value
  Overestimation and Underestimation
  This paper proposes a reinforcement learning framework to enhance the
exploration-exploitation trade-off by learning a range of policies concerning
various confidence bounds. The underestimated values provide stable updates but
suffer from inefficient exploration behaviors. On the other hand, overestimated
values can help the agent escape local optima, but it might cause
over-exploration on low-value areas and function approximation errors
accumulation. Algorithms have been proposed to mitigate the above
contradiction. However, we lack an understanding of how the value bias impact
performance and a method for efficient exploration while keeping value away
from catastrophic overestimation bias accumulation. In this paper, we 1)
highlight that both under- and overestimation bias can improve learning
efficiency, and it is a particular form of the exploration-exploitation
dilemma; 2) propose a unified framework called Realistic Actor-Critic(RAC),
which employs Universal Value Function Approximators (UVFA) to simultaneously
learn policies with different value confidence-bond with the same neural
network, each with a different under-overestimation trade-off. This allows us
to perform directed exploration without over-exploration using the upper bounds
while still avoiding overestimation using the lower bounds. % 3) propose a
variant of soft Bellman backup, called punished Bellman backup, which provides
fine-granular estimation bias control to train policies efficiently. Through
carefully designed experiments, We empirically verify that RAC achieves 10x
sample efficiency and 25\% performance improvement compared to Soft
Actor-Critic on the most challenging Humanoid environment. All the source codes
are available at \url{https://github.com/ihuhuhu/RAC}.
