An Alternate Policy Gradient Estimator for Softmax Policies
  Policy gradient (PG) estimators are ineffective in dealing with softmax
policies that are sub-optimally saturated, which refers to the situation when
the policy concentrates its probability mass on sub-optimal actions.
Sub-optimal policy saturation may arise from bad policy initialization or
sudden changes in the environment that occur after the policy has already
converged. Current softmax PG estimators require a large number of updates to
overcome policy saturation, which causes low sample efficiency and poor
adaptability to new situations. To mitigate this problem, we propose a novel PG
estimator for softmax policies that utilizes the bias in the critic estimate
and the noise present in the reward signal to escape the saturated regions of
the policy parameter space. Our theoretical analysis and experiments, conducted
on bandits and various reinforcement learning environments, show that this new
estimator is significantly more robust to policy saturation.
