\section{Introduction}
\label{sec:intro}


Reinforcement learning (RL)~\citep{sutton2018reinforcement} provides a powerful framework for solving Markov decision process (MDP) problems.
Although deep RL has achieved promising empirical successes in a variety of applications~\citep{schulman2015high, levine2016end}, how to choose RL algorithms \citep{duan2016benchmarking, kakade2002natural, schulman2015trust, peters2008natural, schulman2017proximal} for a specific task is still not fully understood~\citep{henderson2018deep,rajeswaran2017towards}.
This motivates many recent research efforts on understanding the performances of RL algorithms on simplified benchmarks. 
For example, many control applications lead to MDP formulations with continuous state/action spaces, and hence
the classic Linear Quadratic Regulator (LQR) problem has been revisited as a benchmark for understanding the performance of various model-free or model-based RL algorithms on such MDPs~\citep{pmlr-v80-fazel18a,dean2017sample,malik2018derivative,tu2018gap,dean2018regret,abbasi2011regret,abbasi2018regret,yang2019global}.

Another important class of MDPs involve mixed continuous/discrete state variables \citep{boyan2001exact,toussaint2006probabilistic,guestrin2004solving}.
In this paper, our key point is that the problem of controlling unknown (discrete-time) Markov Jump Linear Systems (MJLS) \citep{costa2006discrete} provides  a simple meaningful benchmark for understanding the performances of policy-based RL algorithms
on MDPs with mixed continuous/discrete variables.
MJLS is an important class of dynamical systems that find many applications in control~\citep{bar1993estimation,fox2011tsp,hamsa2016cdc,Pavlovic2000LearningSL,sworder1999estimation,varga2013ijrnc}, and machine learning \citep{hu2017unified,hu2019characterizing}.
Notice that the state/input matrices of a MJLS are functions of a jump parameter that is typically sampled from a Markov chain with discrete state variables. 
Understanding the performance of various RL methods on the MJLS control problem  can bring many useful insights.
An important fact is that the state of a MJLS does not form a Markov chain and hence one cannot study the MJLS control problem as a MDP with solely continuous state variable. 
However, if one augments the state and the jump parameter together, a MDP with a mixed continuous/discrete state space is naturally obtained.
Therefore, we believe the optimal control of unknown MJLS is a meaningful benchmark for further understanding of mixed MDPs.


Although the MDP formulation for MJLSs involves mixed continuous/discrete state variables, our study demonstrates that we can still modify policy-based RL algorithms, such as REINFORCE \citep{williams1992simple,sutton2000policy}, to efficiently solve this problem.
The key here is to parameterize the control policy based on MJLS control theory.
Recently, model-based policy optimization methods have been shown to provably converge to the global optimal policy for MJLS control~\citep{joaoACC}. 
In this paper, we discuss how to efficiently implement these methods in a data-driven manner.
We implement the model-free natural policy gradient (NPG) method on various MJLS examples. 
Our simulation results suggest that the NPG method with the REINFORCE policy gradient estimator and a simple average baseline can learn the optimal control for an MJLS without identifying either the system dynamics or the transition probability of the jump parameter. 
This confirms that policy optimization may provide a promising solution for controlling unknown jump systems. 
It is our hope that our study serves a first step towards more understanding of RL algorithms on MDPs with mixed continuous/discrete variables.

\paragraph{Related work.}
Some previous work has studied how to apply RL methods to MJLSs with known state/input matrices and unknown jump parameter distribution~\citep{costa2002monte, beirigo2018online}. 
In our paper, both state/input matrices and the transition probability of the jump parameter are assumed to be unknown.
In addition, the continuous-time setup has also been investigated recently~\citep{he2019reinforcement}.
Our paper considers the standard discrete-time formulation of MJLSs.
{"config": "arxiv", "file": "2006.03116/intro.tex"}