Regression-aware decompositions
  Linear least-squares regression with a "design" matrix A approximates a given
matrix B via minimization of the spectral- or Frobenius-norm discrepancy
||AX-B|| over every conformingly sized matrix X. Another popular approximation
is low-rank approximation via principal component analysis (PCA) -- which is
essentially singular value decomposition (SVD) -- or interpolative
decomposition (ID). Classically, PCA/SVD and ID operate solely with the matrix
B being approximated, not supervised by any auxiliary matrix A. However, linear
least-squares regression models can inform the ID, yielding regression-aware
ID. As a bonus, this provides an interpretation as regression-aware PCA for a
kind of canonical correlation analysis between A and B. The regression-aware
decompositions effectively enable supervision to inform classical
dimensionality reduction, which classically has been totally unsupervised. The
regression-aware decompositions reveal the structure inherent in B that is
relevant to regression against A.
