Orthogonal Sparse PCA and Covariance Estimation via Procrustes
  Reformulation
  The problem of estimating sparse eigenvectors of a symmetric matrix attracts
a lot of attention in many applications, especially those with high dimensional
data set. While classical eigenvectors can be obtained as the solution of a
maximization problem, existing approaches formulate this problem by adding a
penalty term into the objective function that encourages a sparse solution.
However, the resulting methods achieve sparsity at the expense of sacrificing
the orthogonality property. In this paper, we develop a new method to estimate
dominant sparse eigenvectors without trading off their orthogonality. The
problem is highly non-convex and hard to handle. We apply the MM framework
where we iteratively maximize a tight lower bound (surrogate function) of the
objective function over the Stiefel manifold. The inner maximization problem
turns out to be a rectangular Procrustes problem, which has a closed form
solution. In addition, we propose a method to improve the covariance estimation
problem when its underlying eigenvectors are known to be sparse. We use the
eigenvalue decomposition of the covariance matrix to formulate an optimization
problem where we impose sparsity on the corresponding eigenvectors. Numerical
experiments show that the proposed eigenvector extraction algorithm matches or
outperforms existing algorithms in terms of support recovery and explained
variance, while the covariance estimation algorithms improve significantly the
sample covariance estimator.
