Blending LSTMs into CNNs
  We consider whether deep convolutional networks (CNNs) can represent decision
functions with similar accuracy as recurrent networks such as LSTMs. First, we
show that a deep CNN with an architecture inspired by the models recently
introduced in image recognition can yield better accuracy than previous
convolutional and LSTM networks on the standard 309h Switchboard automatic
speech recognition task. Then we show that even more accurate CNNs can be
trained under the guidance of LSTMs using a variant of model compression, which
we call model blending because the teacher and student models are similar in
complexity but different in inductive bias. Blending further improves the
accuracy of our CNN, yielding a computationally efficient model of accuracy
higher than any of the other individual models. Examining the effect of "dark
knowledge" in this model compression task, we find that less than 1% of the
highest probability labels are needed for accurate model compression.
