Improving Neural Network Robustness via Persistency of Excitation
  Improving adversarial robustness of neural networks remains a major
challenge. Fundamentally, training a neural network via gradient descent is a
parameter estimation problem. In adaptive control, maintaining persistency of
excitation (PoE) is integral to ensuring convergence of parameter estimates in
dynamical systems to their true values. We show that parameter estimation with
gradient descent can be modeled as a sampling of an adaptive linear
time-varying continuous system. Leveraging this model, and with inspiration
from Model-Reference Adaptive Control (MRAC), we prove a sufficient condition
to constrain gradient descent updates to reference persistently excited
trajectories converging to the true parameters. The sufficient condition is
achieved when the learning rate is less than the inverse of the Lipschitz
constant of the gradient of loss function. We provide an efficient technique
for estimating the corresponding Lipschitz constant in practice using extreme
value theory. Our experimental results in both standard and adversarial
training illustrate that networks trained with the PoE-motivated learning rate
schedule have similar clean accuracy but are significantly more robust to
adversarial attacks than models trained using current state-of-the-art
heuristics.
