Robust Implicit Networks via Non-Euclidean Contractions
  Implicit neural networks, a.k.a., deep equilibrium networks, are a class of
implicit-depth learning models where function evaluation is performed by
solving a fixed point equation. They generalize classic feedforward models and
are equivalent to infinite-depth weight-tied feedforward networks. While
implicit models show improved accuracy and significant reduction in memory
consumption, they can suffer from ill-posedness and convergence instability.
  This paper provides a new framework, which we call Non-Euclidean Monotone
Operator Network (NEMON), to design well-posed and robust implicit neural
networks based upon contraction theory for the non-Euclidean norm
$\ell_{\infty}$. Our framework includes (i) a novel condition for
well-posedness based on one-sided Lipschitz constants, (ii) an average
iteration for computing fixed-points, and (iii) explicit estimates on
input-output Lipschitz constants. Additionally, we design a training problem
with the well-posedness condition and the average iteration as constraints and,
to achieve robust models, with the input-output Lipschitz constant as a
regularizer. Our $\ell_{\infty}$ well-posedness condition leads to a larger
polytopic training search space than existing conditions and our average
iteration enjoys accelerated convergence. Finally, we evaluate our framework in
image classification through the MNIST and the CIFAR-10 datasets. Our numerical
results demonstrate improved accuracy and robustness of the implicit models
with smaller input-output Lipschitz bounds. Code is available at
https://github.com/davydovalexander/Non-Euclidean_Mon_Op_Net.
