Differentiable Forward and Backward Fixed-Point Iteration Layers
  Recently, several studies proposed methods to utilize some classes of
optimization problems in designing deep neural networks to encode constraints
that conventional layers cannot capture. However, these methods are still in
their infancy and require special treatments, such as analyzing the KKT
condition, for deriving the backpropagation formula. In this paper, we propose
a new layer formulation called the fixed-point iteration (FPI) layer that
facilitates the use of more complicated operations in deep networks. The
backward FPI layer is also proposed for backpropagation, which is motivated by
the recurrent back-propagation (RBP) algorithm. But in contrast to RBP, the
backward FPI layer yields the gradient by a small network module without an
explicit calculation of the Jacobian. In actual applications, both the forward
and backward FPI layers can be treated as nodes in the computational graphs.
All components in the proposed method are implemented at a high level of
abstraction, which allows efficient higher-order differentiations on the nodes.
In addition, we present two practical methods of the FPI layer, FPI_NN and
FPI_GD, where the update operations of FPI are a small neural network module
and a single gradient descent step based on a learnable cost function,
respectively. FPI\_NN is intuitive, simple, and fast to train, while FPI_GD can
be used for efficient training of energy networks that have been recently
studied. While RBP and its related studies have not been applied to practical
examples, our experiments show the FPI layer can be successfully applied to
real-world problems such as image denoising, optical flow, and multi-label
classification.
