Logarithmic Regret Bound in Partially Observable Linear Dynamical
  Systems
  We study the problem of system identification and adaptive control in
partially observable linear dynamical systems. Adaptive and closed-loop system
identification is a challenging problem due to correlations introduced in data
collection. In this paper, we present the first model estimation method with
finite-time guarantees in both open and closed-loop system identification.
Deploying this estimation method, we propose adaptive control online learning
(AdaptOn), an efficient reinforcement learning algorithm that adaptively learns
the system dynamics and continuously updates its controller through online
learning steps. AdaptOn estimates the model dynamics by occasionally solving a
linear regression problem through interactions with the environment. Using
policy re-parameterization and the estimated model, AdaptOn constructs
counterfactual loss functions to be used for updating the controller through
online gradient descent. Over time, AdaptOn improves its model estimates and
obtains more accurate gradient updates to improve the controller. We show that
AdaptOn achieves a regret upper bound of $\text{polylog}\left(T\right)$, after
$T$ time steps of agent-environment interaction. To the best of our knowledge,
AdaptOn is the first algorithm that achieves $\text{polylog}\left(T\right)$
regret in adaptive control of unknown partially observable linear dynamical
systems which includes linear quadratic Gaussian (LQG) control.
