Linear algebra with transformers
  Most applications of transformers to mathematics, from integration to theorem
proving, focus on symbolic computation. In this paper, we show that
transformers can be trained to perform numerical calculations with high
accuracy. We consider problems of linear algebra: matrix transposition,
addition, multiplication, eigenvalues and vectors, singular value
decomposition, and inversion. Training small transformers (up to six layers)
over datasets of random matrices, we achieve high accuracies (over 90%) on all
problems. We also show that trained models can generalize out of their training
distribution, and that out-of-domain accuracy can be greatly improved by
working from more diverse datasets (in particular, by training from matrices
with non-independent and identically distributed coefficients). Finally, we
show that few-shot learning can be leveraged to re-train models to solve larger
problems.
