Recursive Least Squares Policy Control with Echo State Network
  The echo state network (ESN) is a special type of recurrent neural networks
for processing the time-series dataset. However, limited by the strong
correlation among sequential samples of the agent, ESN-based policy control
algorithms are difficult to use the recursive least squares (RLS) algorithm to
update the ESN's parameters. To solve this problem, we propose two novel policy
control algorithms, ESNRLS-Q and ESNRLS-Sarsa. Firstly, to reduce the
correlation of training samples, we use the leaky integrator ESN and the
mini-batch learning mode. Secondly, to make RLS suitable for training ESN in
mini-batch mode, we present a new mean-approximation method for updating the
RLS correlation matrix. Thirdly, to prevent ESN from over-fitting, we use the
L1 regularization technique. Lastly, to prevent the target state-action value
from overestimation, we employ the Mellowmax method. Simulation results show
that our algorithms have good convergence performance.
