Attending to Mathematical Language with Transformers
  Mathematical expressions were generated, evaluated and used to train neural
network models based on the transformer architecture. The expressions and their
targets were analyzed as a character-level sequence transduction task in which
the encoder and decoder are built on attention mechanisms. Three models were
trained to understand and evaluate symbolic variables and expressions in
mathematics: (1) the self-attentive and feed-forward transformer without
recurrence or convolution, (2) the universal transformer with recurrence, and
(3) the adaptive universal transformer with recurrence and adaptive computation
time. The models respectively achieved test accuracies as high as 76.1%, 78.8%
and 84.9% in evaluating the expressions to match the target values. For the
cases inferred incorrectly, the results differed from the targets by only one
or two characters. The models notably learned to add, subtract and multiply
both positive and negative decimal numbers of variable digits assigned to
symbolic variables.
