Reinforcement learning for port-Hamiltonian systems
  Passivity-based control (PBC) for port-Hamiltonian systems provides an
intuitive way of achieving stabilization by rendering a system passive with
respect to a desired storage function. However, in most instances the control
law is obtained without any performance considerations and it has to be
calculated by solving a complex partial differential equation (PDE). In order
to address these issues we introduce a reinforcement learning approach into the
energy-balancing passivity-based control (EB-PBC) method, which is a form of
PBC in which the closed-loop energy is equal to the difference between the
stored and supplied energies. We propose a technique to parameterize EB-PBC
that preserves the systems's PDE matching conditions, does not require the
specification of a global desired Hamiltonian, includes performance criteria,
and is robust to extra non-linearities such as control input saturation. The
parameters of the control law are found using actor-critic reinforcement
learning, enabling learning near-optimal control policies satisfying a desired
closed-loop energy landscape. The advantages are that near-optimal controllers
can be generated using standard energy shaping techniques and that the
solutions learned can be interpreted in terms of energy shaping and damping
injection, which makes it possible to numerically assess stability using
passivity theory. From the reinforcement learning perspective, our proposal
allows for the class of port-Hamiltonian systems to be incorporated in the
actor-critic framework, speeding up the learning thanks to the resulting
parameterization of the policy. The method has been successfully applied to the
pendulum swing-up problem in simulations and real-life experiments.
