Structural Extensions of Basis Pursuit: Guarantees on Adversarial Robustness
While deep neural networks are sensitive to adversarial noise, sparse coding
using the Basis Pursuit (BP) method is robust against such attacks, including
its multi-layer extensions. We prove that the stability theorem of BP holds
upon the following generalizations: (i) the regularization procedure can be
separated into disjoint groups with different weights, (ii) neurons or full
layers may form groups, and (iii) the regularizer takes various generalized
forms of the $\ell_1$ norm. This result provides the proof for the
architectural generalizations of Cazenavette et al. (2021), including (iv) an
approximation of the complete architecture as a shallow sparse coding network.
Due to this approximation, we settled to experimenting with shallow networks
and studied their robustness against the Iterative Fast Gradient Sign Method on
a synthetic dataset and MNIST. We introduce classification based on the
$\ell_2$ norms of the groups and show numerically that it can be accurate and
offers considerable speedups. In this family, linear transformer shows the best
performance. Based on the theoretical results and the numerical simulations, we
highlight numerical matters that may improve performance further.