Theoretical Interpretation of Learned Step Size in Deep-Unfolded
  Gradient Descent
  Deep unfolding is a promising deep-learning technique in which an iterative
algorithm is unrolled to a deep network architecture with trainable parameters.
In the case of gradient descent algorithms, as a result of the training
process, one often observes the acceleration of the convergence speed with
learned non-constant step size parameters whose behavior is not intuitive nor
interpretable from conventional theory. In this paper, we provide a theoretical
interpretation of the learned step size of deep-unfolded gradient descent
(DUGD). We first prove that the training process of DUGD reduces not only the
mean squared error loss but also the spectral radius related to the convergence
rate. Next, we show that minimizing the upper bound of the spectral radius
naturally leads to the Chebyshev step which is a sequence of the step size
based on Chebyshev polynomials. The numerical experiments confirm that the
Chebyshev steps qualitatively reproduce the learned step size parameters in
DUGD, which provides a plausible interpretation of the learned parameters.
Additionally, we show that the Chebyshev steps achieve the lower bound of the
convergence rate for the first-order method in a specific limit without
learning parameters or momentum terms.
