On a novel training algorithm for sequence-to-sequence predictive
  recurrent networks
  Neural networks mapping sequences to sequences (seq2seq) lead to significant
progress in machine translation and speech recognition. Their traditional
architecture includes two recurrent networks (RNs) followed by a linear
predictor. In this manuscript we perform analysis of a corresponding algorithm
and show that the parameters of the RNs of the well trained predictive network
are not independent of each other. Their dependence can be used to
significantly improve the network effectiveness. The traditional seq2seq
algorithms require short term memory of a size proportional to the predicted
sequence length. This requirement is quite difficult to implement in a
neuroscience context. We present a novel memoryless algorithm for seq2seq
predictive networks and compare it to the traditional one in the context of
time series prediction. We show that the new algorithm is more robust and makes
predictions with higher accuracy than the traditional one.
