Boosting LSTM Performance Through Dynamic Precision Selection
  The use of low numerical precision is a fundamental optimization included in
modern accelerators for Deep Neural Networks (DNNs). The number of bits of the
numerical representation is set to the minimum precision that is able to retain
accuracy based on an offline profiling, and it is kept constant for DNN
inference.
  In this work, we explore the use of dynamic precision selection during DNN
inference. We focus on Long Short Term Memory (LSTM) networks, which represent
the state-of-the-art networks for applications such as machine translation and
speech recognition. Unlike conventional DNNs, LSTM networks remember
information from previous evaluations by storing data in the LSTM cell state.
Our key observation is that the cell state determines the amount of precision
required: time steps where the cell state changes significantly require higher
precision, whereas time steps where the cell state is stable can be computed
with lower precision without any loss in accuracy.
  Based on this observation, we implement a novel hardware scheme that tracks
the evolution of the elements in the LSTM cell state and dynamically selects
the appropriate precision in each time step. For a set of popular LSTM
networks, our scheme selects the lowest precision for more than 66% of the
time, outperforming systems that fix the precision statically. We evaluate our
proposal on top of a modern accelerator highly optimized for LSTM computation,
and show that it provides 1.56x speedup and 23% energy savings on average
without any loss in accuracy. The extra hardware to determine the appropriate
precision represents a small area overhead of 8.8%.
