ACERAC: Efficient reinforcement learning in fine time discretization
  We propose a framework for reinforcement learning (RL) in fine time
discretization and a learning algorithm in this framework. One of the main
goals of RL is to provide a way for physical machines to learn optimal behavior
instead of being programmed. However, the machines are usually controlled in
fine time discretization. The most common RL methods apply independent random
elements to each action, which is not suitable in that setting. It is not
feasible because it causes the controlled system to jerk, and does not ensure
sufficient exploration since a single action is not long enough to create a
significant experience that could be translated into policy improvement. In the
RL framework introduced in this paper, policies are considered that produce
actions based on states and random elements autocorrelated in subsequent time
instants. The RL algorithm introduced here approximately optimizes such a
policy. The efficiency of this algorithm is verified against three other RL
methods (PPO, SAC, ACER) in four simulated learning control problems (Ant,
HalfCheetah, Hopper, and Walker2D) in diverse time discretization. The
algorithm introduced here outperforms the competitors in most cases considered.
