Streaming Linear System Identification with Reverse Experience Replay
  We consider the problem of estimating a linear time-invariant (LTI) dynamical
system from a single trajectory via streaming algorithms, which is encountered
in several applications including reinforcement learning (RL) and time-series
analysis. While the LTI system estimation problem is well-studied in the {\em
offline} setting, the practically important streaming/online setting has
received little attention. Standard streaming methods like stochastic gradient
descent (SGD) are unlikely to work since streaming points can be highly
correlated. In this work, we propose a novel streaming algorithm, SGD with
Reverse Experience Replay ($\mathsf{SGD}-\mathsf{RER}$), that is inspired by
the experience replay (ER) technique popular in the RL literature.
$\mathsf{SGD}-\mathsf{RER}$ divides data into small buffers and runs SGD
backwards on the data stored in the individual buffers. We show that this
algorithm exactly deconstructs the dependency structure and obtains information
theoretically optimal guarantees for both parameter error and prediction error.
Thus, we provide the first -- to the best of our knowledge -- optimal SGD-style
algorithm for the classical problem of linear system identification with a
first order oracle. Furthermore, $\mathsf{SGD}-\mathsf{RER}$ can be applied to
more general settings like sparse LTI identification with known sparsity
pattern, and non-linear dynamical systems. Our work demonstrates that the
knowledge of data dependency structure can aid us in designing statistically
and computationally efficient algorithms which can "decorrelate" streaming
samples.
