Backpropagation Through Time For Networks With Long-Term Dependencies
  Backpropagation through time (BPTT) is a technique of updating tuned
parameters within recurrent neural networks (RNNs). Several attempts at
creating such an algorithm have been made including: Nth Ordered Approximations
and Truncated-BPTT. These methods approximate the backpropagation gradients
under the assumption that the RNN only utilises short-term dependencies. This
is an acceptable assumption to make for the current state of artificial neural
networks. As RNNs become more advanced, a shift towards influence by long-term
dependencies is likely. Thus, a new method for backpropagation is required. We
propose using the 'discrete forward sensitivity equation' and a variant of it
for single and multiple interacting recurrent loops respectively. This solution
is exact and also allows the network's parameters to vary between each
subsequent step, however it does require the computation of a Jacobian.
