A Stable High-order Tuner for General Convex Functions
  Iterative gradient-based algorithms have been increasingly applied for the
training of a broad variety of machine learning models including large
neural-nets. In particular, momentum-based methods, with accelerated learning
guarantees, have received a lot of attention due to their provable guarantees
of fast learning in certain classes of problems and multiple algorithms have
been derived. However, properties for these methods hold only for constant
regressors. When time-varying regressors occur, which is commonplace in dynamic
systems, many of these momentum-based methods cannot guarantee stability.
Recently, a new High-order Tuner (HT) was developed for linear regression
problems and shown to have 1) stability and asymptotic convergence for
time-varying regressors and 2) non-asymptotic accelerated learning guarantees
for constant regressors. In this paper, we extend and discuss the results of
this same HT for general convex loss functions. Through the exploitation of
convexity and smoothness definitions, we establish similar stability and
asymptotic convergence guarantees. Finally, we provide numerical simulations
supporting the satisfactory behavior of the HT algorithm as well as an
accelerated learning property.
