Lattice Recurrent Unit: Improving Convergence and Statistical Efficiency
  for Sequence Modeling
  Recurrent neural networks have shown remarkable success in modeling
sequences. However low resource situations still adversely affect the
generalizability of these models. We introduce a new family of models, called
Lattice Recurrent Units (LRU), to address the challenge of learning deep
multi-layer recurrent models with limited resources. LRU models achieve this
goal by creating distinct (but coupled) flow of information inside the units: a
first flow along time dimension and a second flow along depth dimension. It
also offers a symmetry in how information can flow horizontally and vertically.
We analyze the effects of decoupling three different components of our LRU
model: Reset Gate, Update Gate and Projected State. We evaluate this family on
new LRU models on computational convergence rates and statistical efficiency.
Our experiments are performed on four publicly-available datasets, comparing
with Grid-LSTM and Recurrent Highway networks. Our results show that LRU has
better empirical computational convergence rates and statistical efficiency
values, along with learning more accurate language models.
