AgFlow: Fast Model Selection of Penalized PCA via Implicit
  Regularization Effects of Gradient Flow
  Principal component analysis (PCA) has been widely used as an effective
technique for feature extraction and dimension reduction. In the High Dimension
Low Sample Size (HDLSS) setting, one may prefer modified principal components,
with penalized loadings, and automated penalty selection by implementing model
selection among these different models with varying penalties. The earlier work
[1, 2] has proposed penalized PCA, indicating the feasibility of model
selection in $L_2$- penalized PCA through the solution path of Ridge
regression, however, it is extremely time-consuming because of the intensive
calculation of matrix inverse. In this paper, we propose a fast model selection
method for penalized PCA, named Approximated Gradient Flow (AgFlow), which
lowers the computation complexity through incorporating the implicit
regularization effect introduced by (stochastic) gradient flow [3, 4] and
obtains the complete solution path of $L_2$-penalized PCA under varying
$L_2$-regularization. We perform extensive experiments on real-world datasets.
AgFlow outperforms existing methods (Oja [5], Power [6], and Shamir [7] and the
vanilla Ridge estimators) in terms of computation costs.
