Learning Longer-term Dependencies via Grouped Distributor Unit
  Learning long-term dependencies still remains difficult for recurrent neural
networks (RNNs) despite their success in sequence modeling recently. In this
paper, we propose a novel gated RNN structure, which contains only one gate.
Hidden states in the proposed grouped distributor unit (GDU) are partitioned
into groups. For each group, the proportion of memory to be overwritten in each
state transition is limited to a constant and is adaptively distributed to each
group member. In other word, every separate group has a fixed overall update
rate, yet all units are allowed to have different paces. Information is
therefore forced to be latched in a flexible way, which helps the model to
capture long-term dependencies in data. Besides having a simpler structure, GDU
is demonstrated experimentally to outperform LSTM and GRU on tasks including
both pathological problems and natural data set.
