On Asymptotic Linear Convergence of Projected Gradient Descent for
  Constrained Least Squares
  Many recent problems in signal processing and machine learning such as
compressed sensing, image restoration, matrix/tensor recovery, and non-negative
matrix factorization can be cast as constrained optimization. Projected
gradient descent is a simple yet efficient method for solving such constrained
optimization problems. Local convergence analysis furthers our understanding of
its asymptotic behavior near the solution, offering sharper bounds on the
convergence rate compared to global convergence analysis. However, local
guarantees often appear scattered in problem-specific areas of machine learning
and signal processing. This manuscript presents a unified framework for the
local convergence analysis of projected gradient descent in the context of
constrained least squares. The proposed analysis offers insights into pivotal
local convergence properties such as the condition of linear convergence, the
region of convergence, the exact asymptotic rate of convergence, and the bound
on the number of iterations needed to reach a certain level of accuracy. To
demonstrate the applicability of the proposed approach, we present a recipe for
the convergence analysis of PGD and demonstrate it via a beginning-to-end
application of the recipe on four fundamental problems, namely, linearly
constrained least squares, sparse recovery, least squares with the unit norm
constraint, and matrix completion.
