Auto-Differentiating Linear Algebra
  Development systems for deep learning (DL), such as Theano, Torch,
TensorFlow, or MXNet, are easy-to-use tools for creating complex neural network
models. Since gradient computations are automatically baked in, and execution
is mapped to high performance hardware, these models can be trained end-to-end
on large amounts of data. However, it is currently not easy to implement many
basic machine learning primitives in these systems (such as Gaussian processes,
least squares estimation, principal components analysis, Kalman smoothing),
mainly because they lack efficient support of linear algebra primitives as
differentiable operators. We detail how a number of matrix decompositions
(Cholesky, LQ, symmetric eigen) can be implemented as differentiable operators.
We have implemented these primitives in MXNet, running on CPU and GPU in single
and double precision. We sketch use cases of these new operators, learning
Gaussian process and Bayesian linear regression models, where we demonstrate
very substantial reductions in implementation complexity and running time
compared to previous codes. Our MXNet extension allows end-to-end learning of
hybrid models, which combine deep neural networks (DNNs) with Bayesian
concepts, with applications in advanced Gaussian process models, scalable
Bayesian optimization, and Bayesian active learning.
