What if Neural Networks had SVDs?
  Various Neural Networks employ time-consuming matrix operations like matrix
inversion. Many such matrix operations are faster to compute given the Singular
Value Decomposition (SVD). Previous work allows using the SVD in Neural
Networks without computing it. In theory, the techniques can speed up matrix
operations, however, in practice, they are not fast enough. We present an
algorithm that is fast enough to speed up several matrix operations. The
algorithm increases the degree of parallelism of an underlying matrix
multiplication $H\cdot X$ where $H$ is an orthogonal matrix represented by a
product of Householder matrices. Code is available at
www.github.com/AlexanderMath/fasth .
