Thick-Net: Parallel Network Structure for Sequential Modeling
  Recurrent neural networks have been widely used in sequence learning tasks.
In previous studies, the performance of the model has always been improved by
either wider or deeper structures. However, the former becomes more prone to
overfitting, while the latter is difficult to optimize. In this paper, we
propose a simple new model named Thick-Net, by expanding the network from
another dimension: thickness. Multiple parallel values are obtained via more
sets of parameters in each hidden state, and the maximum value is selected as
the final output among parallel intermediate outputs. Notably, Thick-Net can
efficiently avoid overfitting, and is easier to optimize than the vanilla
structures due to the large dropout affiliated with it. Our model is evaluated
on four sequential tasks including adding problem, permuted sequential MNIST,
text classification and language modeling. The results of these tasks
demonstrate that our model can not only improve accuracy with faster
convergence but also facilitate a better generalization ability.
