Derivative-Free Policy Optimization for Linear Risk-Sensitive and Robust
  Control Design: Implicit Regularization and Sample Complexity
  Direct policy search serves as one of the workhorses in modern reinforcement
learning (RL), and its applications in continuous control tasks have recently
attracted increasing attention. In this work, we investigate the convergence
theory of policy gradient (PG) methods for learning the linear risk-sensitive
and robust controller. In particular, we develop PG methods that can be
implemented in a derivative-free fashion by sampling system trajectories, and
establish both global convergence and sample complexity results in the
solutions of two fundamental settings in risk-sensitive and robust control: the
finite-horizon linear exponential quadratic Gaussian, and the finite-horizon
linear-quadratic disturbance attenuation problems. As a by-product, our results
also provide the first sample complexity for the global convergence of PG
methods on solving zero-sum linear-quadratic dynamic games, a
nonconvex-nonconcave minimax optimization problem that serves as a baseline
setting in multi-agent reinforcement learning (MARL) with continuous spaces.
One feature of our algorithms is that during the learning phase, a certain
level of robustness/risk-sensitivity of the controller is preserved, which we
termed as the implicit regularization property, and is an essential requirement
in safety-critical control systems.
