Neural Programmer: Inducing Latent Programs with Gradient Descent
  Deep neural networks have achieved impressive supervised classification
performance in many tasks including image recognition, speech recognition, and
sequence to sequence learning. However, this success has not been translated to
applications like question answering that may involve complex arithmetic and
logic reasoning. A major limitation of these models is in their inability to
learn even simple arithmetic and logic operations. For example, it has been
shown that neural networks fail to learn to add two binary numbers reliably. In
this work, we propose Neural Programmer, an end-to-end differentiable neural
network augmented with a small set of basic arithmetic and logic operations.
Neural Programmer can call these augmented operations over several steps,
thereby inducing compositional programs that are more complex than the built-in
operations. The model learns from a weak supervision signal which is the result
of execution of the correct program, hence it does not require expensive
annotation of the correct program itself. The decisions of what operations to
call, and what data segments to apply to are inferred by Neural Programmer.
Such decisions, during training, are done in a differentiable fashion so that
the entire network can be trained jointly by gradient descent. We find that
training the model is difficult, but it can be greatly improved by adding
random noise to the gradient. On a fairly complex synthetic table-comprehension
dataset, traditional recurrent networks and attentional models perform poorly
while Neural Programmer typically obtains nearly perfect accuracy.
