Improved Exploring Starts by Kernel Density Estimation-Based State-Space
  Coverage Acceleration in Reinforcement Learning
  Reinforcement learning (RL) is currently a popular research topic in control
engineering and has the potential to make its way to industrial and commercial
applications. Corresponding RL controllers are trained in direct interaction
with the controlled system, rendering them data-driven and performance-oriented
solutions. The best practice of exploring starts (ES) is used by default to
support the learning process via randomly picked initial states. However, this
method might deliver strongly biased results if the system's dynamic and
constraints lead to unfavorable sample distributions in the state space (e.g.,
condensed sample accumulation in certain state-space areas). To overcome this
issue, a kernel density estimation-based state-space coverage acceleration
(DESSCA) is proposed, which improves the ES concept by prioritizing
infrequently visited states for a more balanced coverage of the state space
during training. Compared to neighbouring methods in the field of count-based
exploration, DESSCA can also be applied to continuous state spaces without the
need for artificial discretization of the states. Moreover, the algorithm
allows to define arbitrary reference state distributions such that the state
coverage can be shaped w.r.t. the application needs. Considered test scenarios
are mountain car, cartpole and electric motor control environments. Using DQN
and DDPG as exemplary RL algorithms, it can be shown that DESSCA is a simple
yet effective algorithmic extension to the established ES approach that enables
an increase in learning stability as well as the final control performance.
