Finding Dynamics Preserving Adversarial Winning Tickets
  Modern deep neural networks (DNNs) are vulnerable to adversarial attacks and
adversarial training has been shown to be a promising method for improving the
adversarial robustness of DNNs. Pruning methods have been considered in
adversarial context to reduce model capacity and improve adversarial robustness
simultaneously in training. Existing adversarial pruning methods generally
mimic the classical pruning methods for natural training, which follow the
three-stage 'training-pruning-fine-tuning' pipelines. We observe that such
pruning methods do not necessarily preserve the dynamics of dense networks,
making it potentially hard to be fine-tuned to compensate the accuracy
degradation in pruning. Based on recent works of \textit{Neural Tangent Kernel}
(NTK), we systematically study the dynamics of adversarial training and prove
the existence of trainable sparse sub-network at initialization which can be
trained to be adversarial robust from scratch. This theoretically verifies the
\textit{lottery ticket hypothesis} in adversarial context and we refer such
sub-network structure as \textit{Adversarial Winning Ticket} (AWT). We also
show empirical evidences that AWT preserves the dynamics of adversarial
training and achieve equal performance as dense adversarial training.
