Reliable autonomous systems must account for the uncertainty inherent to the tasks which they aim to solve.
Our theoretical results above indicate that, by using a spectral lines-based approach, one can reliably estimate the parameters of a dynamical system even in the case where deterministic unmodeled dynamics can be present, and have the potential to outperform a Gaussian-noise based input in this task.
The results above hold for large $T$, which may not always be feasible for the data collection of a risk-sensitive autonomous system.
In practice, we still expect the spectral-lines based approach to outperform the approach based on noise, even in the regime with small data.

To this end, we provide experimental evidence which illustrates the efficacy of a spectral lines-based approach in estimating parameters and minimizing risk for a dynamical system.
We show these results in two parts.
First, we highlight the performance of the spectral lines-based approach in estimation of a system of the form \eqref{unmodeled-intro-x}-\eqref{unmodeled-intro-w}, and show that the proper choice of frequencies for spectral lines can lead to significant improvements in parameter estimation.
Next, we show how the use of spectral lines-based exploration in the setting of the Linear Quadratic Regulator can be used achieve low regret in practice.
Our results indicate that while the spectral lines-based approach is comparable to a Gaussian noise-based approach in the system \eqref{lti-noisy-intro}, it outperforms the noise-based approach for the system \eqref{unmodeled-intro-x}-\eqref{unmodeled-intro-w} where the noise in the input can disrupt the system dynamics in unexpected ways.

For both experiments, for simplicity we will use the following structure on the dynamical system.
In both models, we assume the following parameterization of $A_*$ and $B_*$:
\begin{align}
    A_* = \begin{bmatrix}
    0 & 1 & 0 \\
    0 & 0 & 1 \\
    a_1 & a_2 & a_3
    \end{bmatrix}, \qquad B_* = \begin{bmatrix}
    0 \\ 0 \\ 1
    \end{bmatrix} \label{eq:matrices}
\end{align}
As such, these linear components are in control canonical form \cite{Goodwin_1984, Ljung_1987}.
This parameterization of the linear dynamics can be done without loss of generality, in the sense that invertible transformations of the observations can always be made such that the system has the form \eqref{eq:matrices}.
Moreover, this parameterization allows for the spectrum of $A_*$ to be selected in a simple way for each problem setting, as the spectrum will only depend on the three parameters $a_1$, $a_2$, and $a_3$.
The unmodeled dynamics are represented by the following non-linear dynamics:
\begin{align}
    w_{k} &= \begin{bmatrix} 1 & 1 & 1
    \end{bmatrix}^\top \times c \times \overline{w}_k^2, \label{eq:unmodeled-nonlin} \\ \overline{w}_k &= \alpha \overline{w}_{k-1} + \alpha(u_k - \beta u_{k-1}) \label{eq:unmodeled-highpass}
\end{align}
which represent a non-linear transformation in the form of a high pass filter on the control input $u_k$. 
For the simulation results, we use the system defined in \eqref{eq:matrices}-\eqref{eq:unmodeled-highpass} to represent the system \eqref{unmodeled-intro-x}-\eqref{unmodeled-intro-w}.
This choice of parameterization is meant to reflect realistic scenarios present in physical systems, wherein local dynamics are driven mostly by a linear system, but a high frequencies can disrupt the system in unexpected ways.


\subsection{Parameter Estimation with Unmodeled Dynamics}

To approach the task of parameter estimation, we use two types of control inputs in order to estimate the system. 
The first is a white noise-based input, as in Sarkar and Rakhlin \cite{Sarkar_2019}, and is selected as
\begin{equation}
    \label{eq:noise_input}
    u_k = s_k \qquad s_k \sim \mathcal{N}(0, E_0^2) \,,
\end{equation}
The second, a spectral lines-based input, is selected as
\begin{equation}
    \label{eq:pe_input}
    u_k = u_{spec, k} = E_0' \sum_{i = 1}^2 \cos (2 \pi f_i t) \,,
\end{equation}
where $E_0'$ is such that the selection of control inputs ensures $\sum_{k = 1}^T \E\left[ s_k^2 \right] = \sum_{k = 1}^T u_{spec,k}^2 = T E_0$. 
This choice is made to provide a comparison of signals with equivalent energy. 
We first compare the effect of our proposed approach in \eqref{eq:pe_input} with that in \eqref{eq:noise_input} based on parameter estimation, and discuss results based on Regret in the following subsection. 
For each setting, the frequencies $f_i$ are selected based on the problem setting, emulating how practitioners may select frequency content based on the application. 
Details and parameter selections are discussed in Appendix H.

\begin{figure}[!b]
    \centering
    \includegraphics[width = 0.7\textwidth]{imgs/wn_sin_diff.png}
    \caption{Estimation errors for the system \eqref{eq:matrices}-\eqref{eq:unmodeled-highpass}, for various levels $E_0$ of energy of input signals. 
    Frequencies for the spectral-lines based input are $f_1 = 0.01$ and $f_2 = 0.05$, which are selected to be low values due to the knowledge that the unmodeled dynamics reflect a high pass filter. 
    Experiments are run $100$ times, for $T = 500$, and error bars correspond to one standard deviation.
    We see that across all considered choices of the exogeneous input, the lowest estimation error is achieved by the spectral lines-based approach (when $E_0 = 500$), and that at high energy levels the spectral lines-based method consistently outperforms the white noise-based method.}
    \label{fig:exp-results}
\end{figure}

Figure \ref{fig:exp-results} illustrates the estimation errors obtained using \eqref{eq:pe_input} (red) and \eqref{eq:noise_input} (blue) as a function of the input energy $E_0$.
Notably, at higher energies, the spectral-lines based input has both lower estimation error and less variability in the parameter estimation error, as the spectral-lines based input does not excite the unmodeled dynamics.
The experiments also reflect a non-trivial relationship between the energy of the input and the estimation error, as higher energy does not always lead to better estimation when unmodeled dynamics are present.
Rather, in contrast to previous literature which suggests higher levels of exploration noise leads to tighter estimation rates, here we instead see that there is a point at which increased exploration magnitude results in poor estimation due to the exacerbation of the unmodeled dynamics.
Such a point happens for lower levels of energy for the noise-based input, and hence more accurate estimation can be achieved with the spectral lines-based input.

\subsection{Regret of the Linear Quadratic Regulator}

To demonstrate the efficacy of spectral lines-based input with respect to risk minimization of an autonomous system, we apply Algorithm \ref{alg:example} in simulation. 
We first simulate the idealized system in \eqref{lti-noisy-intro}, implemented as in \eqref{eq:matrices} with eigenvalues at 0.9959, 1.01, and 1.0241, which has been considered in recent literature and for which we are able to provide theoretical guarantees on regret.
Figure \ref{fig:regret-results} shows the regret from LQR control of Algorithm \ref{alg:example} as compared to the algorithm for the LQR used in Mania et al. \cite{Mania2019}, which parallels Algorithm \ref{alg:example} except in that the exploration $u_{spec, k}$ is replaced by independent realizations of Gaussian noise.
The results indicate that for the ideal system in \eqref{lti-noisy-intro}, regret is comparable for both approaches.

However, when unmodeled deterministic dynamics are present in the system, as they often are in physical systems, we see that the spectral-lines based exploration outperforms Gaussian noise-based exploration.
Figure \ref{fig:regret-results-unmodeled} highlights this difference, as it shows the regret from LQR control of a more realistic system defined in \eqref{eq:matrices}-\eqref{eq:unmodeled-highpass}. 
As in the previous example, performance is compared between Algorithm \ref{alg:example} as presented, and Algorithm \ref{alg:example} with the modification that $u_{spec, k}$ in line 6 of Algorithm 1 is replaced with $s_k$ from \eqref{eq:noise_input}.
In both experiments, the baseline for calculating regret is an ``optimal'' controller for which $A_*$ and $B_*$ are known and lines 6 and 7 in Algorithm 1 are replaced by $u_k = K_*x_k$, where $K_*$ is the solution to the discrete algebraic Riccati equation. The 

\begin{figure}[!t]
    \centering
    \includegraphics[trim=0 0 0 10, width = 0.7\textwidth]{imgs/regret_n-3/regret_trials-1000_low-noise.png}
    \includegraphics[trim=0 10 0 0, width = 0.7\textwidth]{imgs/regret_n-3/regret_trials-1000_high-noise.png}
    \caption{Median and 90th percentile of regret over time, at two levels of external noise, with no unmodeled dynamics. In both cases, the spectral lines-based exploration (with $f_1 = 0.03$ and $f_2 = 0.05$) has slightly higher median regret than the Gaussian noise-based exploration, but the regrets for both grow at the same rate.}
    \label{fig:regret-results}
\end{figure}

In both simulations, unobserved stochastic noise is present, i.e. $\eta_t$ in \eqref{lti-noisy-intro} and \eqref{unmodeled-intro-x}-\eqref{unmodeled-intro-w} has strictly positive variance.
Each figure contains the results for two noise levels: the top plots use a noise-to-signal ratio of $\sigma = 0.01$, typical in practical applications \cite[Chapter 6.2.1]{Lavretsky2013}, and the bottom plots use a noise-to-signal ratio of $\sigma = 0.1$, to compare to a situation in which noise is larger. 
Additionally, in Figure \ref{fig:regret-results-unmodeled}, the parameters in \eqref{eq:unmodeled-nonlin}-\eqref{eq:unmodeled-highpass} are chosen so that the magnitude of unmodeled dynamics is approximately an order of magnitude smaller than the modeled part at low frequencies.
Figure \ref{fig:regret-results} shows that without unmodeled dynamics, our approach performs comparably to noise-based exploration.
This is unsurprising as there are no high-frequency dynamics for the Gaussian noise to excite. 
However, as discussed in Section 1, unmodeled dynamics are always present for physical systems. 
Figure \ref{fig:regret-results-unmodeled}, then, shows that in a more realistic setting, our approach improves upon the performance of noise-based exploration. 
Additionally, it is worth noting that in our approach, the only noise into the system is the unobserved noise $\eta_k$. Therefore, it is visible in both figures that when the noise-to-signal ratio is small (i.e. in the $\sigma = 0.01$ case), the spectral lines-based exploration has significantly decreased variability in its regret.
\begin{figure}[!t]
    \centering
    \includegraphics[trim=0 0 0 10, width = 0.7\textwidth]{imgs/regret_n-3/regret_trials-1000_low-noise_c-2.png}
    \includegraphics[trim=0 10 0 0, width = 0.7\textwidth]{imgs/regret_n-3/regret_trials-1000_high-noise_c-2.png}
    
    \caption{Median and 90th percentile of Regret over time, at two levels of external noise, with unmodeled dynamics present. In both cases, the spectral lines-based exploration (with $f_1 = 0.03$ and $f_2 = 0.05$) outperforms the Gaussian noise-based exploration.}
    \label{fig:regret-results-unmodeled}
\end{figure}

\subsection{Regret of LQR on Larger Systems}

A larger LTI system ($n = 5$) was also simulated to show the efficacy of our approach over Gaussian noise-based exploration on other systems. The new system took the form of \eqref{eq:matrices-larger}-\eqref{eq:unmodeled-highpass-larger}. For comparability, the eigenvalues were placed at 0.9959, 1.003, 1.01, 1.0171, and 1.0241, at or near the eigenvalues of the system simulated in Section 7.2.
\begin{align}
    A_* &= \begin{bmatrix}
    0 & 1 & 0 & 0 & 0 \\
    0 & 0 & 1 & 0 & 0 \\
    0 & 0 & 0 & 1 & 0 \\
    0 & 0 & 0 & 0 & 1 \\
    a_1 & a_2 & a_3 & a_4 & a_5
    \end{bmatrix}, \qquad B_* = \begin{bmatrix}
    0 \\ 0 \\ 0 \\ 0 \\ 1
    \end{bmatrix} \label{eq:matrices-larger}
\end{align}
\begin{align}
    w_{k} &= \begin{bmatrix} 1 & 1 & 1 & 1 & 1
    \end{bmatrix}^\top \times c \times \overline{w}_k^2, \label{eq:unmodeled-nonlin-larger} \\ \overline{w}_k &= \alpha \overline{w}_{k-1} + \alpha(u_k - \beta u_{k-1}) \label{eq:unmodeled-highpass-larger}
\end{align}
\begin{figure}[!ht]
    \centering
    \includegraphics[trim=0 0 0 10, width = 0.7\textwidth]{imgs/regret_n-5/regret_trials-1000_low-noise.png}
    \includegraphics[trim=0 10 0 0, width = 0.7\textwidth]{imgs/regret_n-5/regret_trials-1000_high-noise.png}
    
    \caption{Simulations of a larger LTI system ($n = 5$). Median and 90th percentile of Regret over time, at two levels of external noise, with no unmodeled dynamics. In both cases, the spectral lines-based exploration with ($f_1 = 0.03$, $f_2 = 0.05$, and $f_3 = 0.07$) has slightly higher median regret than the Gaussian noise-based exploration, but the regrets for both grow at the same rate.}
    \label{fig:regret-results-larger}
\end{figure}

Figure \ref{fig:regret-results-larger} shows the simulation results for the ideal case with no unmodeled dynamics present, i.e. \eqref{lti-noisy-intro} implemented as \eqref{eq:matrices-larger}, and Figure \ref{fig:regret-results-unmodeled-larger} shows the simulation results for the more realistic case with unmodeled dynamics given in \eqref{eq:matrices-larger}-\eqref{eq:unmodeled-highpass-larger}. As in Section 7.2, the top plot of each figure uses a noise-to-signal ratio of 0.01 and the bottom plot uses a ratio of 0.1. The results in Figure \ref{fig:regret-results-larger} show that as in the $n = 3$ case, a deterministic persistently exciting input is comparable to a Gaussian noise-based input without unmodeled dynamics present. Finally, Figure \ref{fig:regret-results-unmodeled-larger} demonstrates that, just as in the $n = 3$ case, our approach outperforms a Gaussian noise-based input when unmodeled dynamics are present.
\begin{figure}[!t]
    \centering
    \includegraphics[trim=0 0 0 10, width = 0.7\textwidth]{imgs/regret_n-5/regret_trials-1000_low-noise_c-2.png}
    \includegraphics[trim=0 10 0 0, width = 0.7\textwidth]{imgs/regret_n-5/regret_trials-1000_high-noise_c-2.png}
    
    \caption{Simulations of a larger LTI system ($n = 5$). Median and 90th percentile of Regret over time, at two levels of external noise, with unmodeled dynamics present. In both cases, the spectral lines-based exploration with ($f_1 = 0.03$, $f_2 = 0.05$, and $f_3 = 0.07$) outperforms the Gaussian noise-based exploration.}
    \label{fig:regret-results-unmodeled-larger}
\end{figure}
{"config": "arxiv", "file": "2006.12687/body/experiments.tex"}