Reinforcement Learning from Demonstrations by Novel Interactive Expert
  and Application to Automatic Berthing Control Systems for Unmanned Surface
  Vessel
  In this paper, two novel practical methods of Reinforcement Learning from
Demonstration (RLfD) are developed and applied to automatic berthing control
systems for Unmanned Surface Vessel. A new expert data generation method,
called Model Predictive Based Expert (MPBE) which combines Model Predictive
Control and Deep Deterministic Policy Gradient, is developed to provide high
quality supervision data for RLfD algorithms. A straightforward RLfD method,
model predictive Deep Deterministic Policy Gradient (MP-DDPG), is firstly
introduced by replacing the RL agent with MPBE to directly interact with the
environment. Then distribution mismatch problem is analyzed for MP-DDPG, and
two techniques that alleviate distribution mismatch are proposed. Furthermore,
another novel RLfD algorithm based on the MP-DDPG, called Self-Guided
Actor-Critic (SGAC) is present, which can effectively leverage MPBE by
continuously querying it to generate high quality expert data online. The
distribution mismatch problem leading to unstable learning process is addressed
by SGAC in a DAgger manner. In addition, theoretical analysis is given to prove
that SGAC algorithm can converge with guaranteed monotonic improvement.
Simulation results verify the effectiveness of MP-DDPG and SGAC to accomplish
the ship berthing control task, and show advantages of SGAC comparing with
other typical reinforcement learning algorithms and MP-DDPG.
