An alternative approach to train neural networks using monotone
  variational inequality
  Despite the vast empirical success of neural networks, theoretical
understanding of the training procedures remains limited, especially in
providing performance guarantees of testing performance due to the non-convex
nature of the optimization problem. The current paper investigates an
alternative approach of neural network training by reducing to another problem
with convex structure -- to solve a monotone variational inequality (MVI) --
inspired by a recent work of (Juditsky & Nemirovsky, 2019). The solution to MVI
can be found by computationally efficient procedures, and importantly, this
leads to performance guarantee of $\ell_2$ and $\ell_{\infty}$ bounds on model
recovery and prediction accuracy under the theoretical setting of training a
single-layer linear neural network. In addition, we study the use of MVI for
training multi-layer neural networks and propose a practical algorithm called
\textit{stochastic variational inequality} (SVI), and demonstrate its
applicability in training fully-connected neural networks and graph neural
networks (GNN) (SVI is completely general and can be used to train other types
of neural networks). We demonstrate the competitive or better performance of
SVI compared to widely-used stochastic gradient descent methods on both
synthetic and real network data prediction tasks regarding various performance
metrics, especially in the improved efficiency in the early stage of training.
