Structured in Space, Randomized in Time: Leveraging Dropout in RNNs for
  Efficient Training
  Recurrent Neural Networks (RNNs), more specifically their Long Short-Term
Memory (LSTM) variants, have been widely used as a deep learning tool for
tackling sequence-based learning tasks in text and speech. Training of such
LSTM applications is computationally intensive due to the recurrent nature of
hidden state computation that repeats for each time step. While sparsity in
Deep Neural Nets has been widely seen as an opportunity for reducing
computation time in both training and inference phases, the usage of non-ReLU
activation in LSTM RNNs renders the opportunities for such dynamic sparsity
associated with neuron activation and gradient values to be limited or
non-existent. In this work, we identify dropout induced sparsity for LSTMs as a
suitable mode of computation reduction. Dropout is a widely used regularization
mechanism, which randomly drops computed neuron values during each iteration of
training. We propose to structure dropout patterns, by dropping out the same
set of physical neurons within a batch, resulting in column (row) level hidden
state sparsity, which are well amenable to computation reduction at run-time in
general-purpose SIMD hardware as well as systolic arrays. We conduct our
experiments for three representative NLP tasks: language modelling on the PTB
dataset, OpenNMT based machine translation using the IWSLT De-En and En-Vi
datasets, and named entity recognition sequence labelling using the CoNLL-2003
shared task. We demonstrate that our proposed approach can be used to translate
dropout-based computation reduction into reduced training time, with
improvement ranging from 1.23x to 1.64x, without sacrificing the target metric.
