Fast Differentiable Matrix Square Root and Inverse Square Root
  Computing the matrix square root and its inverse in a differentiable manner
is important in a variety of computer vision tasks. Previous methods either
adopt the Singular Value Decomposition (SVD) to explicitly factorize the matrix
or use the Newton-Schulz iteration (NS iteration) to derive the approximate
solution. However, both methods are not computationally efficient enough in
either the forward pass or the backward pass. In this paper, we propose two
more efficient variants to compute the differentiable matrix square root and
the inverse square root. For the forward propagation, one method is to use
Matrix Taylor Polynomial (MTP), and the other method is to use Matrix Pad\'e
Approximants (MPA). The backward gradient is computed by iteratively solving
the continuous-time Lyapunov equation using the matrix sign function. A series
of numerical tests show that both methods yield considerable speed-up compared
with the SVD or the NS iteration. Moreover, we validate the effectiveness of
our methods in several real-world applications, including de-correlated batch
normalization, second-order vision transformer, global covariance pooling for
large-scale and fine-grained recognition, attentive covariance pooling for
video recognition, and neural style transfer. The experimental results
demonstrate that our methods can also achieve competitive and even slightly
better performances. The Pytorch implementation is available at
\href{https://github.com/KingJamesSong/FastDifferentiableMatSqrt}{https://github.com/KingJamesSong/FastDifferentiableMatSqrt}.
