SHINE: SHaring the INverse Estimate from the forward pass for bi-level
  optimization and implicit models
  In recent years, implicit deep learning has emerged as a method to increase
the effective depth of deep neural networks. While their training is
memory-efficient, they are still significantly slower to train than their
explicit counterparts. In Deep Equilibrium Models (DEQs), the training is
performed as a bi-level problem, and its computational complexity is partially
driven by the iterative inversion of a huge Jacobian matrix. In this paper, we
propose a novel strategy to tackle this computational bottleneck from which
many bi-level problems suffer. The main idea is to use the quasi-Newton
matrices from the forward pass to efficiently approximate the inverse Jacobian
matrix in the direction needed for the gradient computation. We provide a
theorem that motivates using our method with the original forward algorithms.
In addition, by modifying these forward algorithms, we further provide
theoretical guarantees that our method asymptotically estimates the true
implicit gradient. We empirically study this approach and the recent
Jacobian-Free method in different settings, ranging from hyperparameter
optimization to large Multiscale DEQs (MDEQs) applied to CIFAR and ImageNet.
Both methods reduce significantly the computational cost of the backward pass.
While SHINE has a clear advantage on hyperparameter optimization problems, both
methods attain similar computational performances for larger scale problems
such as MDEQs at the cost of a limited performance drop compared to the
original models.
