Monotone operator equilibrium networks
  Implicit-depth models such as Deep Equilibrium Networks have recently been
shown to match or exceed the performance of traditional deep networks while
being much more memory efficient. However, these models suffer from unstable
convergence to a solution and lack guarantees that a solution exists. On the
other hand, Neural ODEs, another class of implicit-depth models, do guarantee
existence of a unique solution but perform poorly compared with traditional
networks. In this paper, we develop a new class of implicit-depth model based
on the theory of monotone operators, the Monotone Operator Equilibrium Network
(monDEQ). We show the close connection between finding the equilibrium point of
an implicit network and solving a form of monotone operator splitting problem,
which admits efficient solvers with guaranteed, stable convergence. We then
develop a parameterization of the network which ensures that all operators
remain monotone, which guarantees the existence of a unique equilibrium point.
Finally, we show how to instantiate several versions of these models, and
implement the resulting iterative solvers, for structured linear operators such
as multi-scale convolutions. The resulting models vastly outperform the Neural
ODE-based models while also being more computationally efficient. Code is
available at http://github.com/locuslab/monotone_op_net.
