Policy Optimization for Markovian Jump Linear Quadratic Control:
  Gradient-Based Methods and Global Convergence
  Recently, policy optimization for control purposes has received renewed
attention due to the increasing interest in reinforcement learning. In this
paper, we investigate the global convergence of gradient-based policy
optimization methods for quadratic optimal control of discrete-time Markovian
jump linear systems (MJLS). First, we study the optimization landscape of
direct policy optimization for MJLS, with static state feedback controllers and
quadratic performance costs. Despite the non-convexity of the resultant
problem, we are still able to identify several useful properties such as
coercivity, gradient dominance, and almost smoothness. Based on these
properties, we show global convergence of three types of policy optimization
methods: the gradient descent method; the Gauss-Newton method; and the natural
policy gradient method. We prove that all three methods converge to the optimal
state feedback controller for MJLS at a linear rate if initialized at a
controller which is mean-square stabilizing. Some numerical examples are
presented to support the theory. This work brings new insights for
understanding the performance of policy gradient methods on the Markovian jump
linear quadratic control problem.
