"Parallel Training Considered Harmful?": Comparing series-parallel and
  parallel feedforward network training
  Neural network models for dynamic systems can be trained either in parallel
or in series-parallel configurations. Influenced by early arguments, several
papers justify the choice of series-parallel rather than parallel configuration
claiming it has a lower computational cost, better stability properties during
training and provides more accurate results. Other published results, on the
other hand, defend parallel training as being more robust and capable of
yielding more accu- rate long-term predictions. The main contribution of this
paper is to present a study comparing both methods under the same unified
framework. We focus on three aspects: i) robustness of the estimation in the
presence of noise; ii) computational cost; and, iii) convergence. A unifying
mathematical framework and simulation studies show situations where each
training method provides better validation results, being parallel training
better in what is believed to be more realistic scenarios. An example using
measured data seems to reinforce such claim. We also show, with a novel
complexity analysis and numerical examples, that both methods have similar
computational cost, being series series-parallel training, however, more
amenable to parallelization. Some informal discussion about stability and
convergence properties is presented and explored in the examples.
