Band-limited Soft Actor Critic Model
  Soft Actor Critic (SAC) algorithms show remarkable performance in complex
simulated environments. A key element of SAC networks is entropy
regularization, which prevents the SAC actor from optimizing against fine
grained features, oftentimes transient, of the state-action value function.
This results in better sample efficiency during early training. We take this
idea one step further by artificially bandlimiting the target critic spatial
resolution through the addition of a convolutional filter. We derive the closed
form solution in the linear case and show that bandlimiting reduces the
interdependency between the low and high frequency components of the
state-action value approximation, allowing the critic to learn faster. In
experiments, the bandlimited SAC outperformed the classic twin-critic SAC in a
number of Gym environments, and displayed more stability in returns. We derive
novel insights about SAC by adding a stochastic noise disturbance, a technique
that is increasingly being used to learn robust policies that transfer well to
the real world counterparts.
